テキストファイル処理の逆引き辞典的な何か

対象の形式とCLIツール
対象の操作
形式別のまとめ
- plain text
- JSON
- YAML
- XML
- TOML
- CSV
- TSV
- LTSV
- properties

ファイル形式別にCLIツールと基本的な操作を整理したかった。

対象の形式とCLIツール

plain text
- unix tools (sed, grep, awk)
JSON
- ✔github.com/stedolan/jq
- github.com/ahmet2mir/wildq
YAML
XML
TOML
- github.com/kislyuk/yq
- ✔github.com/ahmet2mir/wildq
CSV
- ✔github.com/harelba/q
TSV
- ✔github.com/harelba/q
- github.com/jtolio/tsv-tools
LTSV
- ✔github.com/sters/ltsvq
.properties
- NONE

対象の操作

select/filter
update
delete

形式別のまとめ

plain text

Linuxコマンド逆引き大全 Index:テキスト操作にまとまっててよい。

// https://httpbin.org/json
{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ],
    "title": "Sample Slide Show"
  }
}

select/filter

title に "Wake up" を含む行を選択する。

$ cat httpbin.json | grep -o -P '"title": "Wake up.*"'
"title": "Wake up to WonderWidgets!"

title に "Wake up" を 含まない 行を選択する。 (slideshow.slides[].title だけを抽出するのは難しい)

$ at httpbin.json | grep -o -P '"title": "(?!Wake up).*"'
"title": "Overview"
"title": "Sample Slide Show"

update

s コマンドで slideshow.title を上書きする。

\(\s\{4\}"title"\):\s\{0,1\}"[^"]*" の説明。

パターン	意味
`\(`	置換した後も残す部分のグループを開始
`\s\{4\}`	4文字の空白文字(`U+0020`)にマッチするパターン
`"title"`	見たままの文字列にマッチするパターン
`\)`	置換した後も残す部分のグループを終了
`:`	見たままの文字列にマッチするパターン
`\s\{0,1\}`	0文字以上1文字以下の空白文字(`U+0020`)にマッチするパターン
`"[^"]*"`	ダブルクォート(`U+0022`)で囲まれた、ダブルクォート(`U+0022`)以外の文字列にマッチするパターン

 cat httpbin.json | sed 's/\(\s\{4\}"title"\):\s\{0,1\}"[^"]*"/\1: "Updated Slide Show"/1'
{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Updated Slide Show",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Updated Slide Show",
        "type": "all"
      }
    ],
    "title": "Updated Slide Show"
  }
}

delete

d コマンドで slideshow.title を消す。

前の行の末尾のカンマ(U+002c)も削除しないと構造が壊れてしまう
複数行マッチ機能を使えばどうにかなるのかもしれない

cat httpbin.json | sed '/\s\{2\}"title":.*/d'
{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "type": "all"
      }
    ],
  }
}

merge/override/append

JSON

jq Manualを読み解けばなんでも出来るようになる。

jq playで実験できる。

インストール

Linux/WSL

# Ubuntu/Debian
$ sudo apt-get install jq
# CentOS/RHEL
$ sudo yum install jq
# Binary
$ mkdir -p "${HOME}/bin"
$ curl -fsSL --output "${HOME}/bin/jq" "https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64"
$ chmod +x "${HOME}/bin/jq"

Windows

# Chocolatey
PS> chocolatey install jq
# Scoop
PS> scoop install jq
# Binary
PS> New-Item -Path $env:USERPROFILE -Name bin -ItemType Directory -Force
PS> curl -fsSL --output "$env:USERPROFILE/bin/jq.exe" "https://github.com/stedolan/jq/releases/download/jq-1.6/jq-win64.exe"

// https://httpbin.org/json
{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ],
    "title": "Sample Slide Show"
  }
}

select/filter

select(boolean_expression)

title に "Wake up" を含むオブジェクトを選択する。

# slideshow.slides.title contains "Wake up"
$ cat httpbin.json | jq -r '.slideshow.slides[] | select(.title | contains("Wake up")) | .'
{
  "title": "Wake up to WonderWidgets!",
  "type": "all"
}

title に "Wake up" を 含まない オブジェクトを選択する。

# slideshow.slides.title not contains "Wake up"
$ cat httpbin.json | jq -r '.slideshow.slides[] | select(.title | contains("Wake up") | not) | .'
{
  "items": [
    "Why <em>WonderWidgets</em> are great",
    "Who <em>buys</em> WonderWidgets"
  ],
  "title": "Overview",
  "type": "all"
}

update

Addition:+

+ は上書きする（もとの slideshow が上書きされた）。

$ cat httpbin.json | jq -r '. + {slideshow: {title: "Updated Slide Show"}}'
{
  "slideshow": {
    "title": "Updated Slide Show"
  }
}

Multiplication, division, modulo: *, /, and %

* はマージする（もとの slideshow に新しい slideshow がマージされた）。

$ cat httpbin.json | jq -r '. * {slideshow: {title: "Updated Slide Show"}}'
{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ],
    "title": "Updated Slide Show"
  }
}

delete

演算子はないから代わりに新しいオブジェクトを作る。（元のオブジェクトから author を消した）

$ cat httpbin.json | \
jq -r '.slideshow | {slideshow: { date: .date, title: .title, slides: .slides }}'
{
  "slideshow": {
    "date": "date of publication",
    "title": "Sample Slide Show",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ]
  }
}

YAML

インストール

Linux/WSL

# Binary
$ mkdir -p "${HOME}/bin"
$ durl=$(curl -fsSL "https://api.github.com/repos/mikefarah/yq/releases/latest" | \
jq -r '.assets[] | select( .name | contains("linux_amd64")) | .browser_download_url')
$ curl -fsSL --output "${HOME}/bin/yq" "${durl}"
$ chmod +x "${HOME}/bin/yq"

Windows

# Chocolatey
PS> chocolatey install yq
# Scoop
PS> scoop install yq
# Binary
PS> New-Item -Path $env:USERPROFILE -Name bin -ItemType Directory -Force
PS> $durl = curl -fsSL "https://api.github.com/repos/mikefarah/yq/releases/latest" | `
ConvertFrom-Json | `
Select -Expand assets | `
Where -Match -Property name -Value windows_amd64.exe | `
Select -ExpandProperty browser_download_url
PS> curl -fsSL --output "$env:USERPROFILE/bin/yq.exe" $durl

# httpbin.yaml
slideshow:
  author: Yours Truly
  date: date of publication
  slides:
    - title: Wake up to WonderWidgets!
      type: all
    - items:
        - Why <em>WonderWidgets</em> are great
        - Who <em>buys</em> WonderWidgets
      title: Overview
      type: all
  title: Sample Slide Show

select/filter

Select - yq

title が "Wake up to WonderWidgets!" と 一致する オブジェクトを選択する。

$ yq e '.slideshow.slides[] | select(.title == "Wake up to WonderWidgets!") | .' httpbin.yaml
title: Wake up to WonderWidgets!
type: all

title が "Wake up to WonderWidgets!" と 一致しない オブジェクトを選択する。

$ yq e '.slideshow.slides[] | select(.title != "Wake up to WonderWidgets!") | .' httpbin.yaml
items:
  - Why <em>WonderWidgets</em> are great
  - Who <em>buys</em> WonderWidgets
title: Overview
type: all

update

Reduce - yq

# other.yaml
slideshow:
  title: Updated

同じ構造のドキュメントをマージする。

$ yq eval-all '. as $item ireduce ({}; . * $item)' httpbin.yaml other.yaml
slideshow:
  author: Yours Truly
  date: date of publication
  slides:
    - title: Wake up to WonderWidgets!
      type: all
    - items:
        - Why <em>WonderWidgets</em> are great
        - Who <em>buys</em> WonderWidgets
      title: Overview
      type: all
  title: Updated

delete

Delete - yq

slideshow.author を削除する。

$ yq e 'del(.slideshow.author)' httpbin.yaml
slideshow:
  date: date of publication
  slides:
    - title: Wake up to WonderWidgets!
      type: all
    - items:
        - Why <em>WonderWidgets</em> are great
        - Who <em>buys</em> WonderWidgets
      title: Overview
      type: all
  title: Sample Slide Show

XML

インストール

Linux/WSL

# Ubuntu/Debian
$ apt-get install xmlstarlet
# CentOS/RHEL
$ yum install xmlstarlet

Windows

# Chocolatey
PS> chocolatey install xmlstarlet
# Scoop
PS> scoop install xmlstarlet

<?xml version='1.0' encoding='us-ascii'?>

<!--  A SAMPLE set of slides  -->

<slideshow
    title="Sample Slide Show"
    date="Date of publication"
    author="Yours Truly"
    >

    <!-- TITLE SLIDE -->
    <slide type="all">
      <title>Wake up to WonderWidgets!</title>
    </slide>

    <!-- OVERVIEW -->
    <slide type="all">
        <title>Overview</title>
        <item>Why &gt;em&lt;WonderWidgets&gt;/em&lt; are great</item>
        <item/>
        <item>Who &gt;em&lt;buys&gt;/em&lt; WonderWidgets</item>
    </slide>

</slideshow>

select/filter

1. Querying XML documents

title に "Wake up" を含むオブジェクトを選択する。

$ cat httpbin.xml | MSYS_NO_PATHCONV=1 xml sel -t -m '/slideshow/slide/title[contains(., "Wake up")]' -c ..
<slide type="all">
      <title>Wake up to WonderWidgets!</title>
    </slide>

title に "Wake up" を 含まない オブジェクトを選択する。

$ cat httpbin.xml | MSYS_NO_PATHCONV=1 xml sel -t -m '/slideshow/slide/title[not(contains(., "Wake up"))]' -c ..
<slide type="all">
        <title>Overview</title>
        <item>Why &gt;em&lt;WonderWidgets&gt;/em&lt; are great</item>
        <item/>
        <item>Who &gt;em&lt;buys&gt;/em&lt; WonderWidgets</item>
    </slide>

update

3. Editing XML documents

slideshow[@title] を更新する。

$ cat httpbin.xml | MSYS_NO_PATHCONV=1 xml ed -u '/slideshow/@title' -v "Updated"
<?xml version="1.0" encoding="us-ascii"?>
<!--  A SAMPLE set of slides  -->
<slideshow title="Updated" date="Date of publication" author="Yours Truly">
  <!-- TITLE SLIDE -->
  <slide type="all">
    <title>Wake up to WonderWidgets!</title>
  </slide>
  <!-- OVERVIEW -->
  <slide type="all">
    <title>Overview</title>
    <item>Why &gt;em&lt;WonderWidgets&gt;/em&lt; are great</item>
    <item/>
    <item>Who &gt;em&lt;buys&gt;/em&lt; WonderWidgets</item>
  </slide>
</slideshow>

delete

3. Editing XML documents

slideshow[@author] を削除する。

$ cat httpbin.xml | MSYS_NO_PATHCONV=1 xml ed -d '/slideshow/@author'
<?xml version="1.0" encoding="us-ascii"?>
<!--  A SAMPLE set of slides  -->
<slideshow title="Sample Slide Show" date="Date of publication">
  <!-- TITLE SLIDE -->
  <slide type="all">
    <title>Wake up to WonderWidgets!</title>
  </slide>
  <!-- OVERVIEW -->
  <slide type="all">
    <title>Overview</title>
    <item>Why &gt;em&lt;WonderWidgets&gt;/em&lt; are great</item>
    <item/>
    <item>Who &gt;em&lt;buys&gt;/em&lt; WonderWidgets</item>
  </slide>
</slideshow>

TOML

インストール

Linux(WSL)

# Python
$ pip install wildq
# Ubuntu/Debian
$ durl=$(curl -fsSL "https://api.github.com/repos/ahmet2mir/wildq/releases/latest" | \
jq -r '.assets[] | select( .name | contains(".deb")) | .browser_download_url')
$ curl -fsSL --output "${HOME}/wildq.deb" "${durl}"
$ sudo dpkg -i "${HOME}/wildq.deb"
# CentOS/RHEL
$ durl=$(curl -fsSL "https://api.github.com/repos/ahmet2mir/wildq/releases/latest" | \
jq -r '.assets[] | select( .name | contains(".rpm")) | .browser_download_url')
$ curl -fsSL --output "${HOME}/wildq.rpm" "${durl}"
$ sudo yum install -y "${HOME}/wildq.rpm"
# Binary
$ mkdir -p "${HOME}/bin"
$ durl=$(curl -fsSL "https://api.github.com/repos/ahmet2mir/wildq/releases/latest" | \
jq -r '.assets[] | select( .name | contains("linux-x86_64.tar.gz")) | .browser_download_url')
$ curl -fsSL "${durl}" | tar -xzvf - -C "${HOME}/bin" wq wildq

Windows

# Python
PS> pip install wildq

# httpbin.toml
[slideshow]
author = "Yours Truly"
date = "date of publication"
title = "Sample Slide Show"

[[slideshow.slides]]
title = "Wake up to WonderWidgets!"
type = "all"

[[slideshow.slides]]
items = ["Why <em>WonderWidgets</em> are great", "Who <em>buys</em> WonderWidgets"]
title = "Overview"
type = "all"

select/filter

title に "Wake up" を含むオブジェクトを選択する。

$ cat httpbin.toml | wildq -i toml -r '.slideshow.slides[] | select(.title | contains("Wake up")) | .'
title = "Wake up to WonderWidgets!"
type = "all"

title に "Wake up" を 含まない オブジェクトを選択する。

$ cat httpbin.toml | wildq -i toml -r '.slideshow.slides[] | select(.title | contains("Wake up") | not) | .'
items = [ "Why <em>WonderWidgets</em> are great", "Who <em>buys</em> WonderWidgets",]
title = "Overview"
type = "all"

update

+ は上書きする（もとの slideshow が上書きされた）。

$ $ cat httpbin.toml | wildq -i toml -r '. + {slideshow: {title: "Updated Slide Show"}}'
[slideshow]
title = "Updated Slide Show"

* はマージする（もとの slideshow に新しい slideshow がマージされた）。

$ cat httpbin.toml | wildq -i toml -r '. * {slideshow: {title: "Updated Slide Show"}}'
[slideshow]
author = "Yours Truly"
date = "date of publication"
title = "Updated Slide Show"
[[slideshow.slides]]
title = "Wake up to WonderWidgets!"
type = "all"

[[slideshow.slides]]
items = [ "Why <em>WonderWidgets</em> are great", "Who <em>buys</em> WonderWidgets",]
title = "Overview"
type = "all"

delete

演算子はないから代わりに新しいオブジェクトを作る。（元のオブジェクトから author を消した）

$ cat httpbin.toml | \
wildq -i toml -r '.slideshow | {slideshow: { date: .date, title: .title, slides: .slides }}'
[slideshow]
date = "date of publication"
title = "Sample Slide Show"
[[slideshow.slides]]
title = "Wake up to WonderWidgets!"
type = "all"

[[slideshow.slides]]
items = [ "Why <em>WonderWidgets</em> are great", "Who <em>buys</em> WonderWidgets",]
title = "Overview"
type = "all"

CSV

インストール

Linux/WSL

# Ubuntu/Debian
$ durl=$(curl -fsSL "https://api.github.com/repos/harelba/q/releases/latest" | \
jq -r '.assets[] | select( .name | contains(".deb")) | .browser_download_url')
$ curl -fsSL --output "${HOME}/q.deb" "${durl}"
$ sudo dpkg -i "${HOME}/q.deb"
# CentOS/RHEL
$ durl=$(curl -fsSL "https://api.github.com/repos/harelba/q/releases/latest" | \
jq -r '.assets[] | select( .name | contains(".rpm")) | .browser_download_url')
$ curl -fsSL --output "${HOME}/q.rpm" "${durl}"
$ sudo yum install -y "${HOME}/q.rpm"
# Binary
$ mkdir -p "${HOME}/bin"
$ durl=$(curl -fsSL "https://api.github.com/repos/harelba/q/releases/latest" | \
jq -r '.assets[] | select( .name | contains("-Linux")) | .browser_download_url')
$ curl -fsSL --output "${HOME}/bin/q" "${durl}"
$ chmod +x "${HOME}/bin/q"

Windows

# Binary
PS> New-Item -Path $env:USERPROFILE -Name bin -ItemType Directory -Force
PS> $durl = curl -fsSL "https://api.github.com/repos/harelba/q/releases/latest" | `
ConvertFrom-Json | `
Select -Expand assets | `
Where -Match -Property name -Value Windows.exe | `
Select -ExpandProperty browser_download_url
PS> curl -fsSL --output "$env:USERPROFILE/bin/q.exe" $durl

github.com/inouet/ken-all/testdata/test_001.csv

注意: 文字コードをSJISからUTF-8へ、改行コードをLFへ変換すること

PS> (curl -fsSL https://raw.githubusercontent.com/inouet/ken-all/master/testdata/test_001.csv) -join "`n" | Set-Content -Path test_001.csv -Encoding utf8

PS> q -b -O -d , 'SELECT * FROM test_001.csv'
c1  ,c2,c3    ,c4     ,c5          ,c6                ,c7 ,c8    ,c9         ,c10,c11,c12,c13,c14,c15
1101,60,600000,ﾎｯｶｲﾄﾞｳ,ｻｯﾎﾟﾛｼﾁｭｳｵｳｸ,ｲｶﾆｹｲｻｲｶﾞﾅｲﾊﾞｱｲ   ,北海道,札幌市中央区,以下に掲載がない場合 ,0  ,0  ,0  ,0  ,0  ,0
1101,64,640941,ﾎｯｶｲﾄﾞｳ,ｻｯﾎﾟﾛｼﾁｭｳｵｳｸ,ｱｻﾋｶﾞｵｶ           ,北海道,札幌市中央区,旭ケ丘        ,0  ,0  ,1  ,0  ,0  ,0
1101,60,600041,ﾎｯｶｲﾄﾞｳ,ｻｯﾎﾟﾛｼﾁｭｳｵｳｸ,ｵｵﾄﾞｵﾘﾋｶﾞｼ        ,北海道,札幌市中央区,大通東        ,0  ,0  ,1  ,0  ,0  ,0
1101,60,600042,ﾎｯｶｲﾄﾞｳ,ｻｯﾎﾟﾛｼﾁｭｳｵｳｸ,ｵｵﾄﾞｵﾘﾆｼ(1-19ﾁｮｳﾒ),北海道,札幌市中央区,大通西（１〜１９丁目）,1  ,0  ,1  ,0  ,0  ,0

select/filter

旭ケ丘 を含む行を選択する。

$ q -b -O -d , 'SELECT * FROM test_001.csv WHERE c9="旭ケ丘"'
c1  ,c2,c3    ,c4     ,c5          ,c6     ,c7 ,c8    ,c9 ,c10,c11,c12,c13,c14,c15
1101,64,640941,ﾎｯｶｲﾄﾞｳ,ｻｯﾎﾟﾛｼﾁｭｳｵｳｸ,ｱｻﾋｶﾞｵｶ,北海道,札幌市中央区,旭ケ丘,0  ,0  ,1  ,0  ,0  ,0

旭ケ丘 を 含まない 行を選択する。

$ q -b -O -d , 'SELECT * FROM test_001.csv WHERE c9<>"旭ケ丘"'
c1  ,c2,c3    ,c4     ,c5          ,c6                ,c7 ,c8    ,c9         ,c10,c11,c12,c13,c14,c15
1101,60,600000,ﾎｯｶｲﾄﾞｳ,ｻｯﾎﾟﾛｼﾁｭｳｵｳｸ,ｲｶﾆｹｲｻｲｶﾞﾅｲﾊﾞｱｲ   ,北海道,札幌市中央区,以下に掲載がない場合 ,0  ,0  ,0  ,0  ,0  ,0
1101,60,600041,ﾎｯｶｲﾄﾞｳ,ｻｯﾎﾟﾛｼﾁｭｳｵｳｸ,ｵｵﾄﾞｵﾘﾋｶﾞｼ        ,北海道,札幌市中央区,大通東        ,0  ,0  ,1  ,0  ,0  ,0
1101,60,600042,ﾎｯｶｲﾄﾞｳ,ｻｯﾎﾟﾛｼﾁｭｳｵｳｸ,ｵｵﾄﾞｵﾘﾆｼ(1-19ﾁｮｳﾒ),北海道,札幌市中央区,大通西（１〜１９丁目）,1  ,0  ,1  ,0  ,0  ,0

update

plain text の項を参照。

delete

plain text の項を参照。

TSV

インストール

CSV の項を参照。

github.com/harelba/q/examples/exampledatafile

PS> curl -fsSL --remote-name https://raw.githubusercontent.com/harelba/q/master/examples/exampledatafile

PS> q "SELECT * FROM exampledatafile LIMIT 10"
-rw-r--r-- 1 root root 2064 2006-11-23 21:33 netscsid.conf
-rw-r--r-- 1 root root 1343 2007-01-09 20:39 wodim.conf
-rw-r--r-- 1 root root 112 2007-06-22 18:08 apg.conf
-rw-r--r-- 1 root root 15752 2009-07-25 18:13 ltrace.conf
-rw-r--r-- 1 root root 624 2010-05-16 14:18 mtools.conf
-rw-r--r-- 1 root root 395 2010-06-20 11:11 anacrontab
-rw-r--r-- 1 root root 18673 2010-10-18 06:49 globash.rc
-rw-r--r-- 1 root root 23958 2010-11-15 10:07 mime.types
-rw-r--r-- 1 root root 449 2010-11-15 10:07 mailcap.order
-rw-r--r-- 1 root root 8453 2010-12-03 22:32 nanorc

select/filter

ssh を含む行を選択する。

$ q 'SELECT * FROM exampledatafile WHERE c8="ssh"'
drwxr-xr-x 2 root root 4096 2011-12-22 18:57 ssh

ssh を 含まない 行を選択する。

$ q 'SELECT * FROM exampledatafile WHERE c8<>"ssh" ORDER BY c8 LIMIT 10'
drwxr-xr-x 5 root root 4096 2011-10-12 16:28 ConsoleKit
drwxr-xr-x 5 root root 4096 2011-10-12 16:30 NetworkManager
drwxr-xr-x 2 root root 4096 2011-03-15 23:05 ODBCDataSources
drwxr-xr-x 3 root root 4096 2011-12-18 12:09 OpenCL
drwxr-xr-x 2 root root 4096 2012-01-13 12:48 R
drwxr-xr-x 2 root root 4096 2011-10-12 16:30 UPower
drwxr-xr-x 10 root root 4096 2012-01-16 16:08 X11
drwxr-xr-x 3 root root 4096 2011-10-12 16:30 acpi
-rw-r--r-- 1 root root 2981 2011-10-12 16:27 adduser.conf
-rw-r--r-- 1 root root 10 2011-12-18 11:45 adjtime

update

plain text の項を参照。

delete

plain text の項を参照。

LTSV

インストール

Linux/WSL

# Binary
$ mkdir -p "${HOME}/bin"
$ durl=$(curl -fsSL "https://api.github.com/repos/sters/ltsvq/releases/latest" | \
jq -r '.assets[] | select( .name | contains("linux_amd64.tar.gz")) | .browser_download_url')
$ curl -fsSL "${durl}" | tar -xzvf - -C "${HOME}/bin" ltsvq

Windows

# Binary
# PS> New-Item -Path $env:USERPROFILE -Name bin -ItemType Directory -Force
# PS> $durl = curl -fsSL "https://api.github.com/repos/sters/ltsvq/releases/latest" | `
# ConvertFrom-Json | `
# Select -Expand assets | `
# Where -Match -Property name -Value windows_amd64.tar.gz | `
# Select -ExpandProperty browser_download_url
# PS> curl -fsSL --output "$env:USERPROFILE/ltsvq.tar.gz" $durl
# PS> tar -xzvf "$env:USERPROFILE/ltsvq.tar.gz" -C "$env:USERPROFILE/bin" ltsvq.exe

github.com/sters/ltsvq/example.ltsv

host:127.0.0.1    ident:- user:frank  time:[10/Oct/2000:13:55:36 -0700]   req:GET /apache_pb.gif HTTP/1.0 status:200  size:2326   referer:http://www.example.com/start.html   ua:Mozilla/4.08 [en] (Win98; I ;Nav)
host:192.168.1.1    ident:- user:frank  time:[10/Oct/2000:13:55:36 -0700]   req:GET /apache_pb.gif HTTP/1.0 status:200  size:2326   referer:http://www.example.com/start.html   ua:Mozilla/4.08 [en] (Win98; I ;Nav)
host:127.0.0.1  ident:- user:frank  time:[10/Oct/2000:13:55:36 -0700]   req:GET /apache_pb.gif HTTP/1.0 status:200  size:2326   referer:http://www.example.com/start.html   ua:Mozilla/4.08 [en] (Win98; I ;Nav)
host:192.168.1.1    ident:- user:frank  time:[10/Oct/2000:13:55:36 -0700]   req:GET /apache_pb.gif HTTP/1.0 status:200  size:2326   referer:http://www.example.com/start.html   ua:Mozilla/4.08 [en] (Win98; I ;Nav)
host:127.0.0.1  ident:- user:frank  time:[10/Oct/2000:13:55:36 -0700]   req:GET /apache_pb.gif HTTP/1.0 status:200  size:2326   referer:http://www.example.com/start.html   ua:Mozilla/4.08 [en] (Win98; I ;Nav)
host:192.168.1.1    ident:- user:frank  time:[10/Oct/2000:13:55:36 -0700]   req:GET /apache_pb.gif HTTP/1.0 status:200  size:2326   referer:http://www.example.com/start.html   ua:Mozilla/4.08 [en] (Win98; I ;Nav)
host:127.0.0.1  ident:- user:frank  time:[10/Oct/2000:13:55:36 -0700]   req:GET /apache_pb.gif HTTP/1.0 status:200  size:2326   referer:http://www.example.com/start.html   ua:Mozilla/4.08 [en] (Win98; I ;Nav)
host:192.168.1.1    ident:- user:frank  time:[10/Oct/2000:13:55:36 -0700]   req:GET /apache_pb.gif HTTP/1.0 status:200  size:2326   referer:http://www.example.com/start.html   ua:Mozilla/4.08 [en] (Win98; I ;Nav)

select/filter

host の値が 192. で 始まる 行を選択する。

$ cat example.ltsv | ltsvq -q "select * from ltsv where host like '192%'"
_dummy: host:192.168.1.1        ident:- referer:http://www.example.com/start.html       req:GET /apache_pb.gif HTTP/1.0 size:2326       status:200      time:[10/Oct/2000:13:55:36 -0700]       ua:Mozilla/4.08 [en] (Win98; I ;Nav)     user:frank
_dummy: host:192.168.1.1        ident:- referer:http://www.example.com/start.html       req:GET /apache_pb.gif HTTP/1.0 size:2326       status:200      time:[10/Oct/2000:13:55:36 -0700]       ua:Mozilla/4.08 [en] (Win98; I ;Nav)     user:frank
_dummy: host:192.168.1.1        ident:- referer:http://www.example.com/start.html       req:GET /apache_pb.gif HTTP/1.0 size:2326       status:200      time:[10/Oct/2000:13:55:36 -0700]       ua:Mozilla/4.08 [en] (Win98; I ;Nav)     user:frank
_dummy: host:192.168.1.1        ident:- referer:http://www.example.com/start.html       req:GET /apache_pb.gif HTTP/1.0 size:2326       status:200      time:[10/Oct/2000:13:55:36 -0700]       ua:Mozilla/4.08 [en] (Win98; I ;Nav)     user:frank

host の値が 192. で 始まらない 行を選択する。

$ cat example.ltsv | ltsvq -q "select * from ltsv where host not like '192%'"
_dummy: host:127.0.0.1  ident:- referer:http://www.example.com/start.html   req:GET /apache_pb.gif HTTP/1.0 size:2326   status:200  time:[10/Oct/2000:13:55:36 -0700]   ua:Mozilla/4.08 [en] (Win98; I ;Nav)    user:frank  
_dummy: host:127.0.0.1  ident:- referer:http://www.example.com/start.html   req:GET /apache_pb.gif HTTP/1.0 size:2326   status:200  time:[10/Oct/2000:13:55:36 -0700]   ua:Mozilla/4.08 [en] (Win98; I ;Nav)    user:frank  
_dummy: host:127.0.0.1  ident:- referer:http://www.example.com/start.html   req:GET /apache_pb.gif HTTP/1.0 size:2326   status:200  time:[10/Oct/2000:13:55:36 -0700]   ua:Mozilla/4.08 [en] (Win98; I ;Nav)    user:frank  
_dummy: host:127.0.0.1  ident:- referer:http://www.example.com/start.html   req:GET /apache_pb.gif HTTP/1.0 size:2326   status:200  time:[10/Oct/2000:13:55:36 -0700]   ua:Mozilla/4.08 [en] (Win98; I ;Nav)    user:frank

update

plain text の項を参照。

delete

plain text の項を参照。

properties

TBD