Compare commits

..

1 Commits

Author SHA1 Message Date
Vincent Riquer
1d395757b8 Delete transogg 2025-01-28 02:00:00 +00:00
65 changed files with 1002 additions and 1943 deletions

1
.gitignore vendored
View File

@ -2,4 +2,3 @@
*.EX
trace.log
.vscode/
Makefile.in

View File

@ -1,65 +0,0 @@
# 1.0.6
### BUGS
* Support for newline character in filenames
* Fix obsolete files deletion
### BUGS (Minor)
* Fix timing information
* Fix level 3+ debug levels (hanging waiting for child processes)
# 1.0.5
### BUGS (Minor)
* `toys/createindex`: handle empty channel count, bitdepth and sampling rate cleanly
### Enhancements
* Allow ignoring microsecond precision in timestamps
* Don't print useless information (stuff that handled 0 files and such)
* Store transcoded file paths relative to their destination's root (db version 7)
* Add missing (used but not declared) error codes
# 1.0.4
## `BREAKING CHANGES`
* ffmpeg parser was fixed. All files previously parsed by it and parsers depending on it will be reprocessed. Running in batches (`-B <batchsize>`) is recommended.
### BUGS
* `ffmpeg` output parsed incorrectly
* `toys/createindex`: incorrect path for ID3v1 tag index file
### Enhancements
* Use bash builtin `printf` instead of command `date`
* Commit to database more frequently when reading tags
# 1.0.3
## `BREAKING CHANGES`
* Implementing replaygain copy meant bumping versions of every tag parser. All file tags will be read again. Running in batches (`-B <batchsize>`) is recommended.
### Bugs
* Fix a number of issues with `releasecountry` tag
### Enhancements
* Spawn tasks faster when slots are available.
* Copy replaygain tags.
* Add `copy_extension` to setup
# 1.0.2
### Bugs
* Tag reading hangs on vorbis files with embedded images.
* Recreate destination files on releasecountry tag change (bump database schema to 5).
# 1.0.1
## `BREAKING CHANGES`
* Implementing releasecountry meant bumping versions of every tag parser. All file tags will be read again. Running in batches (`-B <batchsize>`) is recommended.
### Enhancements
* Copy releasecountry tag ("MusicBrainz Album Release Country" for MP3).
### Bugs
* Opus tags were not actually parsed due to an issue with the tag reader selector.
### Enhancements
* Fetch releasecountry tag from files
* Add `%{releasecountry}` placeholder in `rename`
* Add `-r` (show release country) to toys/createindex
# 1.0.0
Initial public release

View File

@ -1,19 +0,0 @@
include Makefile.in
all: atom #$(wildcard toys/*) $(wildcard lib/*/*)
mkdir work
cp -r atom toys work
sed -i 's:%LIBDIR%:'$(libdir)':;s:%SHAREDIR%:'$(sharedir)':;s:%DOCDIR%:'$(docdir)':' work/atom work/toys/*
install:
install -m 644 -D -t $(docdir) doc/*
install -m 644 -D README.md $(docdir)/README.md
install -m 644 -D work/toys/README $(docdir)/README.toys
rm work/toys/README
install -m 644 -D -t $(sharedir) share/*
install -d $(libdir)
cp -dpr --no-preserve=ownership lib/* $(libdir)
install -D -t $(bindir) work/atom work/toys/*
clean:
rm -Rf work

View File

@ -1,38 +1,9 @@
# AtOM: Anything to Ogg and Mp3
This program is for you if
1. you want to have your music collection in the highest quality possible
2. you have devices that don't have enough storage to hold it or
3. you have devices that support a limited number of formats
To satisfy that need, we take a "source" directory, inventory and read tags for
every file therein. From that, we can create and maintain copies of said
directory in other formats or quality. There is also the possibility of
ignoring or simply copying files with certain mime-types or extensions.
We try to do this in a *smart* way, by only treating files that are new or
changed since the last run.
Apart from transcoding from one format to another, AtOM can also change
sample-rate, bitrate, and the number of channels! Say, for example, that you
want to stream your music through icecast. In addition to having all the files
in the same format, it will want a constant sample-rate and bitrate. You can
have AtOM do that!
Here's what I have for my tests:
| Directory | Format | Sample rate | Bitrate | Channels | FAT32 compat. | ASCII | Size |
| --------- | ------ | ----------- | --------- | -------- | ------------- | ----- | ---- |
| 0-Full | Mixed | Mixed | Mixed | Mixed | No | No | 568G |
| 1-High | Vorbis | Same | Quality 5 | Same | Yes | No | 143G |
| 2-Medium | Opus | Same | 64 | Same | Yes | No | 60G |
| 3-Small | Opus | Same | 32 | Same | Yes | No | 31G |
| 4-MP3 | MP3 | 44100 | 128 | 2 | Yes | Yes | 119G |
* URL: https://framagit.org/atom/AtOM/
* URL: https://framagit.org/ScriptFanix/AtOM/
* Author: Vincent Riquer <vincent+prog.atom@riquer.fr>
* Copyright/left: 2012-2013,2015,2025-2026 Vincent Riquer - GPLv3
* Copyright/left: 2012-2013,2015,2025 Vincent Riquer - GPLv3
- except: transogg: WTFPL 2.0
## Dependencies
### Required:
@ -41,7 +12,7 @@ Here's what I have for my tests:
* [SQLite](http://www.sqlite.org/)
### Optional:
**Some features will be disabled** when not present.
Some features will not be available.
* [vorbis-tools](http://www.vorbis.com/)
* `ogginfo` (Ogg Vorbis metadata)
* `oggenc` (Ogg Vorbis encoding)
@ -61,13 +32,6 @@ Here's what I have for my tests:
## Using the software
### Installation
1. Clone the repository: `git clone https://framagit.org/atom/AtOM.git`
2. run `./configure && make && sudo make install`
`./configure` takes an optional prefix: `./configure --prefix=/usr`. Defaults to /usr/local.
### Configuration:
On first run, AtOM will ask a set of questions to help you create a
configuration file.
@ -78,7 +42,7 @@ If, however, you still want to make changes manually, please read doc/config.
There are a lot of comments in the generated config file too.
### Preparing data:
Nothing specific needs to be done. You can edit your tags, rename files, move
Nothing specific needs to be done. You can edit ypur tags, rename files, move
them around how you see fit. However, make sure you setup your tag editor
to *do* update the files' timestamps: though it was initially plan to make this
optional, using checksums or tags, it was abandoned due to the huge amount of

5
TODO Normal file
View File

@ -0,0 +1,5 @@
Tag Guessing
------------
From a user-defined pattern, guess tags for unsupported formats/untagged files
from the file path and file name.

463
atom
View File

@ -1,17 +1,15 @@
#!/usr/bin/env bash
declare -r \
DOCDIR=%DOCDIR% \
LIBDIR=%LIBDIR% \
SHAREDIR=%SHAREDIR%
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
## Define exit codes
source "$SHAREDIR"/errorcodes
# General config errors [10-19]
ELOAD=10
EINTERVAL=11
ENOCFG=19
# Source cofig errors [20-29]
# Destination config errors [30-49]
EFORMAT=30
ECHANNEL=31
EFMTINVPARM=49
# config structures
declare -A \
@ -20,7 +18,6 @@ declare -A \
destinationchannels \
destinationfat32compat \
destinationcopymime \
destinationcopyext \
destinationformat \
destinationfrequency \
destinationid \
@ -38,33 +35,36 @@ declare -A \
exit $EBASHVERS
}
declare -r \
DOCDIR=./doc \
LIBDIR=./lib \
SHAREDIR=./share
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
cffile="$HOME/.atom/atom.cfg"
LC_ALL=C
shopt -s extglob
source "$SHAREDIR"/id3genres
source $SHAREDIR/id3genres
for function in "$LIBDIR"/*/*
do
source "$function"
done
if ! [[ -f "${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg" ]] \
&& [[ -f "$HOME/.atom/atom.cfg" ]]
then
echo "Configuration found in legacy location $HOME/.atom/atom.cfg."\
"Migrating configuration and data to XDG standard"
xdgMigrate
fi
cffile="${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg"
help() {
cat <<-EOF
Options:
-c <file> Load configuration file <file>
-C Dump configuration and exit
-l <load> Override max-load
-f <workers> Use exactly <workers> child processes
-f <workers> Use exactly <workers> child processes
-T <seconds> override load-interval
-F <destination> Force re-generation of all files in
<destination>
@ -77,6 +77,7 @@ help() {
}
#parse arguments
OPTERR=0
while getopts ':c:Cl:T:F:f:B:ShDq' opt
do
case $opt in
@ -117,12 +118,12 @@ do
:)
echo "-$OPTARG requires an argument"
help
exit $EINVARG
exit 127
;;
*)
echo "Unrecognized option: -$OPTARG"
help
exit $EINVARG
exit 127
;;
esac
done
@ -175,7 +176,163 @@ set +H
(( cfgdump )) && exit
# check sanity
sanityCheck
if [ ! -d "$tempdir" ] && ! mkdir -p "$tempdir"
then
echo "[FATAL] Could not create temp directory $tempdir" >&2
(( sanityfail++ ))
fi
if [ ! -f "$database" ] && [ ! -d "${database%/*}" ] && ! mkdir -p "${database%/*}"
then
echo "[FATAL] Directory holding database file does not exist and could" \
"not be created" >&2
(( sanityfail++ ))
fi
if [ ! -d "$sourcepath" ]
then
echo "[FATAL] Source path $sourcepath does not exist or is not a directory" >&2
(( sanityfail++ ))
fi
if ! which sed >/dev/null
then
echo "[FATAL] Required tool sed is not installed or not in PATH
I never thought this would actually hit someone..." >&2
(( sanityfail++ ))
fi
if ! which sox >/dev/null
then
echo "[FATAL] Required tool sox is not installed or not in PATH" >&2
(( sanityfail++ ))
fi
if ! which ogginfo >/dev/null
then
echo "[WARNING] Tool ogginfo (from vorbis-tools) is not" \
"installed or not in PATH
WebM metadata disabled" >&2
disableogginfo=1
(( sanitywarn++ ))
fi
if ! which soxi >/dev/null
then
echo "[WARNING] Tool soxi (from sox) is not" \
"installed or not in PATH
Vorbis metadata disabled" >&2
disablesoxi=1
(( sanitywarn++ ))
fi
if (( oggencneeded )) && ! which oggenc >/dev/null
then
echo "[WARNING] Tool oggenc (from vorbis-tools) is not" \
"installed or not in PATH
Vorbis targets disabled" >&2
disableoggenc=1
(( sanitywarn++ ))
fi
if ! which opusinfo >/dev/null
then
echo "[WARNING] Tool opusinfo (from opus-tools) is not" \
"installed or not in PATH
Opus metadata disabled" >&2
disableopusinfo=1
(( sanitywarn++ ))
fi
if (( opusencneeded )) && ! which opusenc >/dev/null
then
echo "[WARNING] Tool opusenc (from opus-tools) is not" \
"installed or not in PATH
Opus targets disabled" >&2
disableopusenc=1
(( sanitywarn++ ))
fi
if ! which opusdec >/dev/null
then
echo "[WARNING] Tool opusdec (from opus-tools) is not" \
"installed or not in PATH
Opus support disabled" >&2
disableopusdec=1
(( sanitywarn++ ))
fi
if (( lameneeded )) && ! which lame >/dev/null
then
echo "[WARNING] Tool lame is not installed or not in PATH
MP3 targets disabled" >&2
disablelame=1
(( sanitywarn++ ))
fi
if ! which metaflac >/dev/null
then
echo "[WARNING] Tool metaflac (from FLAC) is not installed" \
"or not in PATH
FLAC metadata disabled" >&2
disableflac=1
(( sanitywarn++ ))
fi
if ! which mpcdec >/dev/null
then
echo "[WARNING] Tool mpcdec (from Musepack) is not" \
"installed or not in PATH
Musepack support disabled" >&2
disablempcdec=1
(( sanitywarn++ ))
fi
if ! which mkvextract >/dev/null
then
echo "[WARNING] Tool mkvextract (from MKVToolNix) is not" \
"installed or not in PATH
WebM metadata disabled
WebM support disabled" >&2
disablemkvextract=1
(( sanitywarn++ ))
fi
if ! which ffprobe >/dev/null
then
echo "[WARNING] Tool ffprobe (from FFmpeg) is not installed or not in PATH
Video metadata disabled
MPEG metadata disabled
MusePack metadata disabled
Unknown format metadata disabled" >&2
disableffprobe=1
(( sanitywarn++ ))
fi
if ! which ffmpeg >/dev/null
then
echo "[WARNING] Tool ffmpeg is not installed or not in PATH
Video support disabled" >&2
disablevideo=1
(( sanitywarn++ ))
fi
if (( textunidecodeneeded )) && ! perl -MText::Unidecode -e 'exit;' 2>/dev/null
then
echo "[WARNING] Perl module Text::Unidecode is not available
Renaming to ASCII-only disabled" >&2
unset destinationascii
destinationascii=0
textunidecodeneeded=0
(( sanitywarn++ ))
fi
if (( sanityfail ))
then
echo "
Sanity checks raised ${sanitywarn:-0} warnings, $sanityfail failures. Dying now." >&2
exit $ESANITY
elif (( sanitywarn ))
then
echo "
Sanity checks raised $sanitywarn warnings... Hit Control-C to abort." >&2
if ! (( cron ))
then
timeout=$(( sanitywarn * 10 ))
echo -n "Starting in $(printf %3i $timeout)" \
$'seconds...\b\b\b\b\b\b\b\b\b\b\b' >&2
while (( timeout ))
do
echo -n $'\b\b\b'"$(printf %3i $timeout)" >&2
sleep 1
(( timeout-- ))
done
echo -en "\r\033[K"
fi
fi
openDatabase
for destination in "${destinations[@]}"
@ -202,44 +359,36 @@ echo '
FROM destination_files
WHERE source_file_id is NULL;' >&3
read -u4 -r -d $'\0' removecount
read -u4 removecount
until (( ${#removefile[@]} == removecount ))
do
echo '
SELECT destination_files.id,
destinations.name,
destination_files.filename
SELECT id,
filename
FROM destination_files
INNER JOIN destinations
ON destination_files.destination_id
= destinations.id
WHERE source_file_id is NULL
LIMIT 500 OFFSET '${#removefile[@]}';
SELECT "AtOM:NoMoreFiles";
' >&3
read -u4 -r -d $'\0' line
read -u4 line
until [[ $line == AtOM:NoMoreFiles ]]
do
removeFileId=${line%%::AtOM:SQL:Sep::*}
rest=${line#*::AtOM:SQL:Sep::}
removeFileDestName=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
removefile[$removeFileId]="${destinationpath["$removeFileDestName"]}/${rest%%::AtOM:SQL:Sep::*}"
read -u4 -r -d $'\0' line
removefile[${line%::AtOM:SQL:Sep::*}]="${line#*::AtOM:SQL:Sep::}"
read -u4 line
done
done
unset deleted
unset removed
deleted=0
removed=0
echo 'BEGIN TRANSACTION;' >&3
for id in ${!removefile[@]}
do
filename=${removefile[id]}
if [ -n "$filename" ]
then
if rm "$filename"
if rm -f "$filename"
then
Delete destination_files <<<"id = $id"
(( ++deleted ))
@ -256,9 +405,9 @@ do
done
echo 'COMMIT;' >&3
(( cron )) || echo -n $'\r'
echo -n "${deleted+$deleted files deleted${removed:+, }}${removed:+$removed removed from database}"
echo -n "Suppressed $deleted files, $removed removed from database"
(( cron )) || echo -ne "\033[K"
(( deleted || removed )) && echo
echo
unset removecount deleted removed removefile
updateTags
@ -272,8 +421,7 @@ do
Update destination_files last_change 0 \
<<<"destination_id = $forcedestid"
else
echo "Full rebuild of destination $forcedest was requested," \
"but it does not exist!" >&2
echo "Destination $forcedest does not exist!" >&2
fi
done
@ -281,7 +429,7 @@ echo '
CREATE TEMPORARY TABLE tasks(
id INTEGER PRIMARY KEY,
requires INTEGER,
required_by INTEGER DEFAULT 0,
required INTEGER,
status INTEGER NOT NULL,
key TEXT UNIQUE,
rename_pattern TEXT,
@ -289,7 +437,6 @@ echo '
ascii INTEGER,
source_file INTEGER,
fileid INTEGER,
destdir TEXT,
filename TEXT,
cmd_arg0 TEXT,
cmd_arg1 TEXT,
@ -387,7 +534,7 @@ echo '
<> CAST(source_files.last_change AS TEXT)
AND mime_type_actions.destination_id = destinations.id
AND mime_type_actions.action = 1;' >&3
read -u4 -r -d $'\0' filecount
read -u4 filecount
if [ -n "$maxbatch" ] && (( maxbatch < filecount ))
then
(( togo = filecount - maxbatch ))
@ -400,23 +547,20 @@ echo '
mime_type_actions.mime_text,
destinations.name,
destination_files.id,
tags.album,
tags.albumartist,
tags.artist,
tags.bitrate,
tags.channels,
tags.composer,
tags.depth,
tags.disc,
tags.genre,
tags.performer,
tags.rate,
tags.releasecountry,
tags.replaygain_alb,
tags.replaygain_trk,
tags.title,
tags.channels,
tags.bitrate,
tags.genre,
tags.albumartist,
tags.year,
tags.album,
tags.disc,
tags.artist,
tags.track,
tags.year
tags.title,
tags.composer,
tags.performer
FROM source_files
INNER JOIN destination_files
ON source_files.id
@ -436,13 +580,13 @@ echo '
(( maxbatch )) && echo "LIMIT $maxbatch" >&3
echo ';
SELECT "AtOM:NoMoreFiles";' >&3
read -u4 -r -d $'\0' line
read -u4 line
while ! [[ $line = AtOM:NoMoreFiles ]]
do
decodefiles+=("$line::AtOM:SQL:Sep::")
read -u4 -r -d $'\0' line
read -u4 line
done
(( cron )) || echo -n $'Creating tasks...\033[K'
(( cron )) || echo -n 'Creating tasks... '
(( textunidecodeneeded )) && ascii
@ -459,39 +603,33 @@ do
rest=${rest#*::AtOM:SQL:Sep::}
destfileid=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
album=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
albumartist=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
artist=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
bitrate=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
channels=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
composer=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
depth=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
disc=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
genre=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
performer=${rest%%::AtOM:SQL:Sep::*}
bitdepth=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
rate=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
releasecountry=${rest%%::AtOM:SQL:Sep::*}
channels=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
replaygain_alb=${rest%%::AtOM:SQL:Sep::*}
bitrate=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
replaygain_trk=${rest%%::AtOM:SQL:Sep::*}
genre=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
title=${rest%%::AtOM:SQL:Sep::*}
albumartist=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
year=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
album=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
disc=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
artist=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
track=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
year=${rest%%::AtOM:SQL:Sep::*}
title=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
composer=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
performer=${rest%%::AtOM:SQL:Sep::*}
unset rest
case ${destinationformat["$destination"]} in
vorbis) (( disableoggenc )) && continue ;;
@ -503,7 +641,7 @@ do
getDestFile
for copy_ext in "${destinationcopyext[@]}"
do
if [[ $filename =~ '.*\.'"$copy_ext"'$' ]]
if [[ $filename =~ '.*\.'$copy_ext'$' ]]
then
copied=1
break
@ -519,13 +657,13 @@ do
album \
albumartist \
artist \
bitdepth \
bitrate \
channels \
commandline \
composer \
copied \
decodetaskid \
depth \
destfileid \
destination \
disc \
@ -534,9 +672,6 @@ do
mimetype \
performer \
rate \
releasecountry \
replaygain_alb \
replaygain_trk \
rest \
sox_needed \
soxoptions_in \
@ -548,11 +683,8 @@ do
tmpfile
done
echo 'COMMIT;' >&3
(( cron )) || echo -n $'\r\033[K'
(( count )) \
&& echo "Created $count tasks for $filecount files" \
"${togo:+($togo left) }" \
"${copies:+($copies immediate copies)}"
(( cron )) || echo -n $'\r'
echo "Created ${count:-0} tasks for $filecount files ${togo:+($togo left) }(${copies:-0} immediate copies)"
# remove perl unicode to ascii coprocess
(( textunidecodeneeded )) && eval exec "${toascii[1]}>&-"
@ -560,20 +692,19 @@ echo 'COMMIT;' >&3
concurrency=$(( maxload / 2 ))
(( concurrency )) || concurrency=1
active=0
concurrencychange=$EPOCHSECONDS
concurrencychange=$(date +%s)
starttime=$concurrencychange
taskcount=$count
remaining=$taskcount
failed=0
echo 'BEGIN TRANSACTION;' >&3
committime=$EPOCHSECONDS
while (( remaining || ${#workers[@]} ))
committime=$(date +%s)
while (( (remaining || ${#workers[@]}) && ! quit ))
do
timestamp=$EPOCHSECONDS
if (( $timestamp - committime >= 60 ))
if (( $(date +%s) - committime >= 60 ))
then
echo $'COMMIT;\nBEGIN TRANSACTION;' >&3
committime=$timestamp
committime=$(date +%s)
fi
read humanload garbage < /proc/loadavg
load=${humanload%.*}
@ -581,28 +712,34 @@ do
then
concurrency="$fixed_workers"
else
if (( timestamp - concurrencychange >= loadinterval ))
if [ -z "$quit" ] \
&& (( ! pause )) \
&& (( $(date +%s)-concurrencychange >= loadinterval ))
then
if (( concurrency > 1 || allow_zero_running )) \
&& (( load > maxload && concurrency ))
if (( concurrency > 1 )) \
&& (( load > maxload ))
then
concurrencychange=$timestamp
concurrencychange=$(date +%s)
(( --concurrency ))
elif (( load < maxload )) \
&& (( active > concurrency - 1 ))
elif (( load < maxload )) && (( active > concurrency - 1 ))
then
concurrencychange=$timestamp
concurrencychange=$(date +%s)
(( ++concurrency ))
fi
fi
fi
checkworkers
cleaner
master
(( pause )) || master
if (( ran - failed ))
then
currenttime=$timestamp
(( runtime = currenttime - starttime ))
currenttime=$(date +%s)
if (( pause ))
then
(( runtime = pausestart - starttime - pausedtime ))
else
(( runtime = currenttime - starttime - pausedtime ))
fi
avgduration=$((
( runtime * 1000)
/
@ -638,7 +775,11 @@ do
fmtprogress="T:%${#taskcount}i/%i (F:%i) %3i%%"
fmttime='%2id %2ih%02im%02is (A:%4.1fs/task)'
eta="ETA:$(
printf "%(%c)T" "$(( currenttime + secsremaining ))"
date -d "${days:-0} days
${hours:-0} hours
${minutes:-0} minutes
${seconds:-0} seconds" \
+'%d/%m %H:%M:%S'
)"
(( cron )) || printf \
"\r$fmtload $fmtworkers $fmtprogress $fmttime $eta\033[K"\
@ -655,7 +796,7 @@ do
${minutes:-0} \
${seconds:-0} \
${avgdsec:-0}.${avgdmsec:-0}
if ! (( concurrency )) && ! (( cron ))
if (( pause ))
then
if (( active ))
then
@ -668,24 +809,24 @@ done
echo 'COMMIT;' >&3
unset count
endtime=$EPOCHSECONDS
endtime=$(date +%s)
(( elapsedseconds = endtime - starttime ))
(( elapsedseconds = endtime - starttime - pausedtime ))
(( days =
elapsedseconds
/
( 24*60*60 )
)) || unset days
)) || true
(( hours =
( elapsedseconds - ( days*24*60*60 ) )
/
( 60*60 )
)) || (( days )) || unset hours
)) || true
(( minutes =
( elapsedseconds - ( ( days*24 + hours ) *60*60 ) )
/
60
)) || (( days || hours )) || unset minutes
)) || true
(( seconds =
elapsedseconds
-
@ -693,15 +834,10 @@ endtime=$EPOCHSECONDS
)) || true
(( cron )) || echo -n $'\r'
(( ran )) \
&& echo -n "Ran $ran tasks${failed:+, $failed of which failed,}" \
"in ${days:+$days days,}" \
"${hours:+$hours hours,}" \
"${minutes:+$minutes minutes and}" \
"$seconds seconds."
echo -n "Ran ${ran:=0} tasks, $failed of which failed, in $days" \
"days, $hours hours, $minutes minutes and $seconds seconds."
(( cron )) || echo -en "\033[K"
(( ran )) && echo
echo
if (( failed ))
then
echo $'\nFailed tasks:\n'
@ -770,24 +906,32 @@ then
FROM tasks
INNER JOIN source_files
ON tasks.source_file=source_files.id
WHERE tasks.status = 2;
WHERE tasks.status = 2
AND requires is NULL;
SELECT "AtOM:NoMoreFiles";' >&3
read -u4 -r -d $'\0' line
read -u4 line
while ! [[ $line = AtOM:NoMoreFiles ]]
do
failedtasks+=("$line")
read -u4 -r -d $'\0' line
read -u4 line
done
for line in "${failedtasks[@]}"
do
echo "${line%%::AtOM:SQL:Sep::*}"
line="${line#*::AtOM:SQL:Sep::}"
line="${line//::AtOM:SQL:Sep::/ }"
echo $'\t'"${line/+( )$/}"$'\n'
echo $'\t'"${line/+( )$/}"
echo
done
fi
if [ -n "$quit" ]
then
closeDatabase
exit
fi
for destination in "${!destinationpath[@]}"
do
echo '
@ -802,7 +946,6 @@ do
tags.disc,
tags.genre,
tags.performer,
tags.releasecountry,
tags.title,
tags.track,
tags.year
@ -833,11 +976,11 @@ do
SELECT "AtOM:NoMoreFiles";
' >&3
read -u4 -r -d $'\0' line
read -u4 line
while [[ $line != AtOM:NoMoreFiles ]]
do
renamefiles+=("$line")
read -u4 -r -d $'\0' line
read -u4 line
done
if (( ${#renamefiles[@]} ))
then
@ -871,8 +1014,6 @@ do
rest=${rest#*::AtOM:SQL:Sep::}
performer=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
releasecountry=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
title=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
track=${rest%%::AtOM:SQL:Sep::*}
@ -887,9 +1028,7 @@ do
progressSpin
if [[ "$oldfilename" != "$destfilename" ]]
then
mv \
"${destinationpath[$destination]}/$oldfilename" \
"${destinationpath[$destination]}/$destfilename"
mv "$oldfilename" "$destfilename"
(( changedcount++ ))
commit=1
fi
@ -901,7 +1040,7 @@ do
"${destinationfat32compat["$destination"]}," \
" ascii=" \
"${destinationascii["$destination"]}" \
"WHERE id=$destfileid;" \
"WHERE id=$destfileid;" \
>&3
if (( commit ))
then
@ -914,10 +1053,9 @@ do
(( textunidecodeneeded )) && eval exec "${toascii[1]}>&-"
echo 'COMMIT;' >&3
(( cron )) || echo -n $'\r'
(( changedcount )) \
&& echo -n "$destination: Renamed $changedcount files"
echo -n "$destination: Renamed ${changedcount:-0} files"
(( cron )) || echo -en "\033[K"
(( changedcount )) && echo
echo
fi
unset count changedcount renamefiles
done
@ -925,26 +1063,22 @@ done
copyFiles_action
echo '
SELECT destination_files.id,
destination_files.filename,
destination_files.old_filename,
destinations.name
SELECT id,
filename,
old_filename
FROM destination_files
INNER JOIN destinations
ON destination_files.destination_id
= destinations.id
WHERE old_filename IS NOT NULL;
SELECT "AtOM:NoMoreFiles";
' >&3
(( cron )) || echo -n 'Removing obsolete files...'$'\033[K'
(( cron )) || echo -n 'Removing obsolete files... '
lines=()
read -u4 -r -d $'\0' line
read -u4 line
while [[ $line != AtOM:NoMoreFiles ]]
do
lines+=("$line")
read -u4 -r -d $'\0' line
read -u4 line
done
echo 'BEGIN TRANSACTION;' >&3
for line in "${lines[@]}"
@ -952,12 +1086,10 @@ do
id=${line%%::AtOM:SQL:Sep::*}
rest=${line#*::AtOM:SQL:Sep::}
filename=${rest%%::AtOM:SQL:Sep::*}
rest=${line#*::AtOM:SQL:Sep::}
oldfilename=${rest%%::AtOM:SQL:Sep::*}
destination=${rest#*::AtOM:SQL:Sep::}
oldfilename=${rest#*::AtOM:SQL:Sep::}
if [[ $oldfilename != "$filename" ]] && [ -f "$oldfilename" ]
then
rm -f "${destinationpath[$destination]}/$oldfilename"
rm -f "$oldfilename"
fi
Update destination_files old_filename NULL <<<"id = $id"
(( count++ ))
@ -965,12 +1097,11 @@ do
done
echo 'COMMIT;' >&3
(( cron )) || echo -n $'\r'
(( count )) \
&& echo -n "Removed $count obsolete files."
echo -n "Removed ${count:-0} obsolete files."
(( cron )) || echo -en "\033[K"
(( count )) && echo
echo
(( debug )) && echo "Purging empty directories..."
echo "Purging empty directories."
for path in "${destinationpath[@]}"
do
find "$path" -type d -empty -delete

28
configure vendored
View File

@ -1,28 +0,0 @@
#!/usr/bin/env bash
# defaults
default_prefix=/usr/local
#default_bindir=$default_prefix/bin
#default_libdir=$default_prefix/lib
#default_sharedir=$default_prefix/share
while (( $# ))
do
case "$1" in
--prefix=*) prefix="${1#*=}"
;;
esac
shift
done
bindir="${prefix:-$default_prefix}"/bin
libdir="${prefix:-$default_prefix}"/lib/AtOM
sharedir="${prefix:-$default_prefix}"/share/AtOM
docdir="${prefix:-$default_prefix}"/share/doc/AtOM
cat > Makefile.in <<-EOMakefile.in
bindir = "$bindir"
libdir = "$libdir"
sharedir = "$sharedir"
docdir = "$docdir"
EOMakefile.in

View File

@ -39,11 +39,6 @@ Sections:
* debug <level>: Integer. Currently defined values:
* 1: few additional status informations.
* 3: log SQL queries.
* skip-timestamp-microsec <bool>: Ignore microsecond precision in timestamps.
Microsec precise timestamps are still stored as-is in the DB, this
setting just impacts comparisons when determining if a file has been
changed.
[source]
This section defines where are the files you want transcoded.

View File

@ -1,189 +1,55 @@
[general]
# This section contains parameters of the program itself.
# * max-load <load>: Integer. Defines how parallel processing will behave. AtOM
# will try to keep the 1 minute load average between <load> and <load>+1 by
# adjusting concurrency.
# Initial concurrency will be set to half of that value.
max-load 16
# * load-interval <seconds>: Integer. How often should we check the load average
# and adjust concurrency. Set this too low, and concurrency may be increased
# too quickly. Set this too high, and AtOM will not adapt quickly enough to
# load increase. In both cases, your hard drive will suffer. In my
# experience, 30 seconds is a good value.
load-interval 10
# * ionice <class> [niceness]: IO-hungry processes will be run with ionice class
# <class> and niceness [niceness] (if applicable). See man ionice for details.
ionice 3
# * temporary-directory <directory>: String. Name speaks for itself: this is
# where FIFOs (for communicating with sqlite) and temporary WAVE files will
# be created. Note that debug logs (if enabled) will go there too.
temporary-directory /tmp/AtOM-user/
# * database <file>: String. Where the SQLite database should be stored.
database /home/user/.local/share/AtOM/atom.db
# * debug <level>: Integer.
# 0: No debug output, encoding and decoding errors logged to
# <temporary-directory>/workerN.log
# 1: Print some debug to stdout, log decoding and encoding commands to
# <temporary-directory>/workerN.log
# 3: log SQL queries to <temporary-directory>/debug.log
#debug 1
# * skip-timestamp-microsec: Ignore microsecond precision in timestamps.
skip-timestamp-microsec 0
max-load 6
load-interval 30
temporary-directory %HOME%/.atom/tmp
database %HOME%/.atom/atom.db
debug 0
[source]
# This section defines where are the files you want transcoded.
path /var/lib/mpd/music
skip /last
skip /lastfm
skip /zzz-atrier
# * path <directory>: String. The root of your collection.
# Default: /var/lib/mpd/music
path /mnt/Musique
# * skip <directory>: String. Files in <directory> will be ignored. Note that
# <directory> can be any expression accepted by find.
skip /lost+found
skip /last
skip /lastfm
skip /zzz-atrier
[Vorbis]
# Each section not named 'general' or 'source' will define a new destination.
# Common parameters:
# Mandatory parameters:
# * enabled: Whether or not to treat this destination (1=true/0=false)
enabled 1
# * path: Where files will be written
path /mnt/Musique-OggQ2
# * format: copy, ogg, opus or mp3. Other formats may appear in the future -
# feel free to implement your preferred format.
format vorbis
# Ogg parameters:
# * quality <quality>: The quality parameter of oggenc. See man oggenc for
# more info. This is the only mode supported and planned. Still, if you want
# to be able to use bitrate settings, feel free to fork and file a pull
# request.
quality 1
# Optional parameters:
# * normalize <yes>/<no>: Normalize output files.
normalize yes
# * rename <string>: Destination files will be named according to <string>,
# after expansion of special strings:
# %{album},
# %{albumartist},
# %{artist},
# %{disc},
# %{genre},
# %{releasecountry},
# %{title},
# %{track},
# %{year}.
# Untagged files or files in unrecognized formats will not be changed.
rename %{genre}/%{albumartist}/%{year}-%{album}-%{releasecountry}/%{disc}%{track}--%{artist}-%{title}
# * fat32compat <yes>/<no>: Rename files for compatibility with FAT32
# filesystems.
fat32compat yes
# * ascii-only <yes>/<no>: Rename files for compatibility with ASCII-only
# systems.
ascii-only no
[Ogg]
path /mnt/Musique-OggQ2
format vorbis
quality 1
normalize yes
channels 2
frequency 44100
# you should not skip or copy application/octet-stream, they could be something
# similar to "Audio file with ID3 version 2.4.0, unsynchronized frames"
# * skip_mime-type <mime-type>: Files with mime-type <mime-type> will not
# be included in that destination. For more than one mime-type, use multiple
# times, as needed. The '*' character is a wildcard.
skip_mime-type inode/x-empty
skip_mime-type audio/midi
skip_mime-type image/*
skip_mime-type text/*
skip_mime-type application/pdf
skip_mime-type application/javascript*
skip_mime-type very short file (no magic)
# * copy_mime-type <mime-type>: Same as skip_mime-type, except that files
# matching will be copied as-is to the destination. E.g. image/* will copy
# covers and other images to the destination. In fact, AtOM will try to use
# hard links instead of copies.
copy_mime-type image/*
# * copy_extension <extension>: Copy files whose name and with ".<extension>"
copy_extension txt
# * channels <number>: Files with more than <number> channels will be
# downmixed. Useful if you create files for telephony music-on-hold.
channels 2
# * frequency <hertz>: Files will be resampled as needed to <hertz>Hz
# sampling-rate. Shoutcast/Icecast streams require a constant sampling-rate.
# Telephony systems often require a sample rate of 8000Hz.
frequency 44100
# * higher-than <bitrate>: Integer. Only reencode files with bitrates higher
# then <bitrate>kbps. This only applies if sample-rate, channel count and of
# course format are equal. If unset, only files with bitrates equal to that
# of the target will be copied (actually, hardlinking will be attempted
# first). As Ogg Vorbis target quality is not defined by its bitrate, Ogg
# Vorbis files will always be reencoded if unset.
higher-than 200
copy_mime-type image/*
copy_mime-type text/*
[Opus]
enabled 1
path /mnt/Musique-opus
format opus
# Opus parameters:
# * bitrate <bitrate>: Set (VBR) bitrate to <bitrate>. Note that while Opus
# allows for decimal values, AtOM does not. The reason for this is simple:
# we do numeric comparisons, and Bash only manipulates integers.
bitrate 96
normalize yes
frequency 48000
copy_mime-type image/*
copy_mime-type text/*
[MP3]
enabled 1
path /mnt/Musique-mp3
path /mnt/Musique-mp3.test
format mp3
# MP3 parameters:
# * bitrate <bitrate>: Set ABR to <bitrate>. Again, if you want CBR or any
# other mode supported by lame, please fork and file a pull request.
bitrate 96
# * noresample <yes>/<no>: LAME may decide to encode your file to a lower
# sampling-rate if you use a low bitrate. Setting this to yes will
# append --resample <original file's rate>, preventing any resampling from
# happening.
noresample yes
normalize yes
higher-than 128
# rename file, path unchanged
rename %{track}--%{artist}-%{title}
# change the whole filepath
#rename %{genre}/%{albumartist}/%{year}-%{album}-%{releasecountry}/%{track}--%{artist}-%{title}
#rename %{genre}/%{albumartist}/%{year}-%{album}/%{track}--%{artist}-%{title}
skip_mime-type image/*
skip_mime-type text/*
[asterisk]
enabled 1
path /mnt/Musique-asterisk
format vorbis
quality 0

View File

@ -5,7 +5,7 @@ getConfigDestination() {
destinationenabled["$destination"]="$value"
;;
'path')
destinationpath["$destination"]="${value%/}"
destinationpath["$destination"]="$value"
;;
'format')
case "$value" in

View File

@ -75,9 +75,6 @@ getConfigGeneral() {
'database')
database="$value"
;;
'skip-timestamp-microsec')
skip_us_timestamp="$value"
;;
debug)
(( value > debug )) && debug=$value
;;

View File

@ -49,13 +49,13 @@ printConfig() {
EOF
[ -n "${destinationskipmime["$destination"]}" ] \
&& echo " |Skipped mime-types|${destinationskipmime["$destination"]//\|/
| |}"
| | |}"
[ -n "${destinationcopymime["$destination"]}" ] \
&& echo " |Copied mime-types|${destinationcopymime["$destination"]//\|/
| |}"
| | |}"
[ -n "${destinationcopyext["$destination"]}" ] \
&& echo " |Copied extensions|${destinationcopyext["$destination"]//\|/
| |}"
| | |}"
done
}|column -t -s'|'
}

View File

@ -33,9 +33,6 @@ database $database
# * debug <level>: Integer.
#debug 1
# * skip-timestamp-microsec: Ignore microsecond precision in timestamps.
skip-timestamp-microsec ${skip_us_timestamp:-0}
[source]
# This section defines where are the files you want transcoded.
@ -63,7 +60,7 @@ path $sourcepath
# Common parameters:
# Mandatory parameters:
# * enabled: Whether or not to treat this destination (1=true/0=false)
# * enabled: Whether or not to treat this destination (1=tue/0=false)
enabled 1
# * path: Where files will be written
@ -147,7 +144,6 @@ bitrate ${destinationquality["$destination"]}
# %{artist},
# %{disc},
# %{genre},
# %{releasecountry},
# %{title},
# %{track},
# %{year}.
@ -211,7 +207,8 @@ bitrate ${destinationquality["$destination"]}
done
cat <<-EOCfg
# * copy_extension <extension>: Copy files whose name and with ".<extension>"
# * copy_extension <extension>: Same as skip_extension, except that files
# matching will be copied as-is to the destination.
EOCfg
destinationcopyext["$destination"]="${destinationcopyext["$destination"]}|"
while [[ ${destinationcopyext["$destination"]} =~ \| ]]

View File

@ -1,6 +1,6 @@
#!/bin/bash
copyFiles_action() {
(( cron )) || echo -n $'Copying files...\033[K'
(( cron )) || echo -n "Copying files... "
echo '
SELECT
source_files.filename,
@ -22,11 +22,11 @@ copyFiles_action() {
AND mime_type_actions.action = 2;
SELECT "AtOM:NoMoreFiles";' >&3
read -u4 -r -d $'\0' line
read -u4 line
while ! [[ $line = AtOM:NoMoreFiles ]]
do
copyfiles+=("$line")
read -u4 -r -d $'\0' line
read -u4 line
done
echo 'BEGIN TRANSACTION;' >&3
@ -79,30 +79,15 @@ copyFiles_action() {
fi
fi
fi
if cp -a --reflink=always \
"$sourcepath/$sourcefilename" \
"$destdir" \
2>/dev/null \
|| cp -al \
"$sourcepath/$sourcefilename" \
"$destdir" \
2>/dev/null \
|| cp -a \
"$sourcepath/$sourcefilename" \
"$destdir"
if cp -al "$sourcepath/$sourcefilename" "$destdir" 2>/dev/null\
|| cp -a "$sourcepath/$sourcefilename" "$destdir"
then
destfilename=${sourcefilename//$'\n'/::AtOM:NewLine:SQL:Inline::}
Update destination_files \
filename \
"$destdir/${destfilename##*/}"\
rename_pattern \
"${destinationrenamepath[$destination]}/${destinationrename[$destination]}"\
fat32compat \
${destinationfat32compat["$destination"]}\
ascii \
${destinationascii["$destination"]}\
last_change \
$lastchange \
filename "$destdir/${sourcefilename##*/}"\
rename_pattern "${destinationrenamepath[$destination]}/${destinationrename[$destination]}"\
fat32compat ${destinationfat32compat["$destination"]}\
ascii ${destinationascii["$destination"]}\
last_change $lastchange \
<<-EOWhere
id = $destfileid
EOWhere
@ -113,12 +98,12 @@ copyFiles_action() {
if (( count ))
then
(( cron )) || echo -n $'\r'
echo -n "Copied ${done:-0} of $count" \
"files${postponed+ ($postponed postponed)}."
echo -n "Copied ${done:-0} of $count" \
"files${postponed+ ($postponed postponed)}."
(( cron )) || echo -en "\033[K"
echo
else
(( cron )) || echo -n $'\r\033[K'
(( cron )) || echo -e "\rNothing to copy.\033[K"
fi
unset count done
}

View File

@ -21,7 +21,7 @@ guessPath() {
LIMIT 1
),"0.0");
'>&3
read -u4 -r -d $'\0' timestamp
read -u4 timestamp
if (( ${timestamp/./} == 0 ))
then
return 2
@ -46,7 +46,7 @@ guessPath() {
LIMIT 1
),"AtOM:NotFound");
'>&3
read -u4 -r -d $'\0' filename
read -u4 filename
if [[ $filename != AtOM:NotFound ]]
then
echo "${filename%/*}"

View File

@ -1,36 +1,33 @@
#!/bin/bash
copyFiles_matching() {
local extension="${filename##*.}"
if \
extension="${filename##*.}"
cp -al \
"$sourcepath/$filename" \
"${destinationpath[$destination]}/$destdir/$destfile.$extension" \
"$destdir/$destfile.$extension" \
2>/dev/null \
|| cp -a \
"$sourcepath/$filename" \
"${destinationpath[$destination]}/$destdir/$destfile.$extension"
then
echo \
"UPDATE destination_files" \
"SET filename=" \
"\"${destdir//\"/\"\"}/${destfile//\"/\"\"}.$extension\"," \
" last_change=(" \
" SELECT last_change" \
" FROM source_files" \
" WHERE id=$fileid" \
" )," \
" old_filename=(" \
" SELECT filename" \
" FROM destination_files" \
" WHERE id=$destfileid" \
" )," \
" rename_pattern=" \
"\"${destinationrenamepath[$destination]}/${destinationrename[$destination]}\","\
" fat32compat=" \
"${destinationfat32compat["$destination"]}," \
" ascii=${destinationascii["$destination"]}"\
"WHERE id=$destfileid;" \
>&3
(( ++copies ))
fi
"$destdir/$destfile.$extension"
echo \
"UPDATE destination_files" \
"SET filename=" \
"\"${destdir//\"/\"\"}/${destfile//\"/\"\"}.$extension\"," \
" last_change=(" \
" SELECT last_change" \
" FROM source_files" \
" WHERE id=$fileid" \
" )," \
" old_filename=(" \
" SELECT filename" \
" FROM destination_files" \
" WHERE id=$destfileid" \
" )," \
" rename_pattern=" \
"\"${destinationrenamepath[$destination]}/${destinationrename[$destination]}\","\
" fat32compat=" \
"${destinationfat32compat["$destination"]}," \
" ascii=${destinationascii["$destination"]}" \
"WHERE id=$destfileid;" \
>&3
(( ++copies ))
}

View File

@ -26,7 +26,6 @@ Insert() {
insert_values+=$value
;;
*)
value=${value//::AtOM:NewLine:SQL:Inline::/$'\n'}
insert_values+='"'"${value//\"/\"\"}"'"'
;;
esac
@ -37,7 +36,7 @@ Insert() {
"( $insert_values );" >&3
(( no_id )) || {
echo 'SELECT LAST_INSERT_ROWID();' >&3
read -u4 -r -d $'\0' results
read -u 4 results
echo "$results"
}
}

View File

@ -19,7 +19,6 @@ Select() {
while read key operator value
do
(( ${#where_statement} )) && where_statement+=( "AND" )
value=${value//::AtOM:NewLine:SQL:Inline::/$'\n'}
where_statement+=( "$key $operator "'"'"${value//\"/\"\"}"'"' )
done
echo "SELECT IFNULL(" \
@ -27,7 +26,7 @@ Select() {
"WHERE ${where_statement[@]})" \
",'SQL::Select:not found'" \
");" >&3
read -u 4 -r -d $'\0' results
read -u 4 results
if ! [[ $results == "SQL::Select:not found" ]]
then
echo "$results"

View File

@ -1,5 +1,5 @@
#!/usr/bin/env bash
currentdbversion=8
#!/bin/bash
currentdbversion=3
checkDatabaseVersion() {
local dbversion
if dbversion=$(Select atom version <<<"\"1\" = 1")
@ -15,10 +15,10 @@ checkDatabaseVersion() {
dbversion=$(Select atom version <<<"\"1\" = 1")
done
else
echo "Database schema version $dbversion is" \
"higher thanthat of this version of" \
"AtOM ($currentdbversion). Bailing out." >&2
exit $EDBVERSION
echo "Database schema version $dbversion is higher than
that of this version of AtOM
($currentdbversion). Bailing out." >&2
exit 1
fi
else
Insert atom 1 <<<"version $currentdbversion"

View File

@ -3,8 +3,7 @@ closeDatabase() {
echo 'vacuum;' >&3
echo .quit >&3
(( debug )) && echo -n "Waiting for SQLite to terminate... "
(( debug > 2 )) && exec 5>&-
wait $db_pid
wait
(( debug )) && echo OK
exec 3>&-
exec 4<&-

View File

@ -1,34 +1,25 @@
#!/usr/bin/env bash
#!/bin/bash
openDatabase() {
local \
populate_db
[[ -f "$database" ]] || populate_db=1
rm -f "$tempdir"/sqlite.{in,out}
mkfifo "$tempdir"/sqlite.{in,out}
stdbuf -o0 sqlite3 -bail \
-newline $'::AtOM:SQL:EOL::\n' \
"$database" \
< "$tempdir/sqlite.in" \
| stdbuf -o0 \
sed 's/::AtOM:SQL:EOL::/\x0/g;s/\(\x0\)\xA/\1/g' \
sqlite3 -bail "$database" \
< "$tempdir/sqlite.in" \
> "$tempdir/sqlite.out" &
db_pid=$!
exec 3> "$tempdir"/sqlite.in
exec 4< "$tempdir"/sqlite.out
rm "$tempdir"/sqlite.{in,out}
rm "$tempdir"/sqlite.in "$tempdir"/sqlite.out
if (( debug > 2 ))
then
exec 5>&3
exec 3> >(tee -a "$tempdir/debug.log" >&5)
exec 3> >(tee -a $tempdir/debug.log >&5)
fi
(( populate_db )) && cat $schema >&3
cat $schema >&3
echo '.separator ::AtOM:SQL:Sep::' >&3
echo 'PRAGMA foreign_keys = ON;' >&3
echo 'PRAGMA recursive_triggers = ON;' >&3
echo 'PRAGMA temp_store = 2;' >&3
echo 'PRAGMA locking_mode = EXCLUSIVE;' >&3
read -u4 -r -d $'\0'
read -u4
unset REPLY
checkDatabaseVersion
}

View File

@ -1,4 +1,4 @@
#!/usr/bin/env bash
#!/bin/bash
upgradedatabase_1_2() {
local data \
@ -18,11 +18,11 @@ FROM destination_files;
SELECT "AtOM::NoMoreData";' >&3
read -u4 -r -d $'\0' data
read -u4 data
while [[ $data != AtOM::NoMoreData ]]
do
datas+=( "$data" )
read -u4 -r -d $'\0' data
read -u4 data
done
echo 'BEGIN TRANSACTION;' >&3
for data in "${datas[@]}"

View File

@ -1,4 +1,4 @@
#!/usr/bin/env bash
#!/bin/bash
upgradedatabase_2_3() {
local data \

View File

@ -1,9 +0,0 @@
#!/usr/bin/env bash
upgradedatabase_3_4() {
echo "Upgrading database to version 4... (backup is $database.bak_v3)"
cp "$database" "$database.bak_v3"
echo 'ALTER TABLE tags ADD COLUMN releasecountry TEXT;' >&3
Update atom version 4 <<<"1 = 1"
}

View File

@ -1,33 +0,0 @@
#!/usr/bin/env bash
upgradedatabase_4_5() {
echo "Upgrading database to version 5... (backup is $database.bak_v4)"
cp "$database" "$database.bak_v4"
echo 'DROP TRIGGER force_destination_update_on_tag_update;' >&3
echo '
CREATE TRIGGER IF NOT EXISTS force_destination_update_on_tag_update
AFTER UPDATE OF
genre,
albumartist,
year,
album,
disc,
artist,
track,
title,
composer,
performer,
releasecountry,
rate,
channels,
bitrate,
depth
ON tags
BEGIN
UPDATE destination_files SET last_change=0
WHERE source_file_id=old.source_file;
END;
' >&3
Update atom version 5 <<<"1 = 1"
}

View File

@ -1,37 +0,0 @@
#!/usr/bin/env bash
upgradedatabase_5_6() {
echo "Upgrading database to version 6... (backup is $database.bak_v5)"
cp "$database" "$database.bak_v5"
echo '
ALTER TABLE tags ADD COLUMN replaygain_alb TEXT;
ALTER TABLE tags ADD COLUMN replaygain_trk TEXT;
DROP TRIGGER force_destination_update_on_tag_update;
CREATE TRIGGER IF NOT EXISTS force_destination_update_on_tag_update
AFTER UPDATE OF
genre,
albumartist,
year,
album,
disc,
artist,
track,
title,
composer,
performer,
releasecountry,
replaygain_alb,
replaygain_trk,
rate,
channels,
bitrate,
depth
ON tags
BEGIN
UPDATE destination_files SET last_change=0
WHERE source_file_id=old.source_file;
END;
' >&3
Update atom version 6 <<<"1 = 1"
}

View File

@ -1,11 +0,0 @@
#!/usr/bin/env bash
upgradedatabase_6_7() {
echo "Upgrading database to version 7... (backup is $database.bak_v6)"
cp "$database" "$database.bak_v6"
for destination in "${destinations[@]}"
do
echo "UPDATE destination_files SET filename = REPLACE(filename,'${destinationpath[$destination]}/','') WHERE filename LIKE '${destinationpath[$destination]}/%';" >&3
done
Update atom version 7 <<<"1 = 1"
}

View File

@ -1,9 +0,0 @@
#!/usr/bin/env bash
upgradedatabase_7_8() {
echo "Upgrading database to version 8... (backup is $database.bak_v7)"
cp "$database" "$database.bak_v7"
echo 'Deletion of old files was failing. Users of previous versions (YOU!) are strongly advised to run cleandestinations with the "-r" flag.'
read -p "Press Enter to continue..."
Update atom version 8 <<<"1 = 1"
}

View File

@ -153,24 +153,18 @@ decodeFile() {
done
)
status 0
cleanup $tmpfile.wav
EOInsert
)
progressSpin
fi
if (( sox_needed ))
then
cleanup="$tempdir/$tmpfile"
decodeSox "$tempdir/$tmpfile.wav"
if ! soxtaskid=$(
Select tasks id <<<"key = $tmpfile"
)
then
parent_required=$(
Select tasks required_by \
<<<"id = $decodetaskid"
)
Update tasks required_by $((++parent_required)) \
<<<"id = $decodetaskid"
soxtaskid=$(
Insert tasks <<-EOInsert
key $tmpfile
@ -182,8 +176,9 @@ decodeFile() {
done
)
requires $decodetaskid
required $decodetaskid
status 0
cleanup $tmpfile.wav
cleanup $cleanup
EOInsert
)
progressSpin

View File

@ -2,5 +2,5 @@
decodeMpcdec() {
tmpfile="${fileid}mpcdec"
commandline=(${ionice}mpcdec)
commandline+=("$sourcepath/${filename//$'\n'/::AtOM:NewLine:SQL:Inline::}" "$tempdir/$tmpfile.wav")
commandline+=("$sourcepath/$filename" "$tempdir/$tmpfile.wav")
}

View File

@ -2,5 +2,5 @@
decodeOpusdec() {
tmpfile="${fileid}opusdec"
commandline=(${ionice}opusdec)
commandline+=("$sourcepath/${filename//$'\n'/::AtOM:NewLine:SQL:Inline::}" "$tempdir/$tmpfile.wav")
commandline+=("$sourcepath/$filename" "$tempdir/$tmpfile.wav")
}

View File

@ -12,7 +12,7 @@ decodeSox() {
then
commandline+=("$1")
else
commandline+=("$sourcepath/${filename//$'\n'/::AtOM:NewLine:SQL:Inline::}")
commandline+=("$sourcepath/$filename")
fi
if [ -n "${destinationfrequency["$destination"]}" ] \
&& (( ${rate:-0} != ${destinationfrequency["$destination"]} ))
@ -26,7 +26,7 @@ decodeSox() {
commandline+=(-c ${destinationchannels["$destination"]})
soxoptions_out+=" -c ${destinationchannels["$destination"]}"
fi
if (( ${depth:-0} > 16 ))
if (( ${bitdepth:-0} > 16 ))
then
commandline+=(-b 16)
soxoptions_out+=" -b 16"

View File

@ -1,6 +1,6 @@
#!/usr/bin/env bash
#!/bin/bash
encodeFile::mp3() {
lameopts=(${ionice}lame --quiet --noreplaygain)
lameopts=(${ionice}lame --quiet)
lameopts+=(-v --abr ${destinationquality[$destination]})
[ -n "$album" ] && lameopts+=(--tl "$album" )
[ -n "$artist" ] && lameopts+=(--ta "$artist")
@ -11,12 +11,6 @@ encodeFile::mp3() {
[ -n "$albumartist" ] && lameopts+=(--tv TPE2="$albumartist")
[ -n "$composer" ] && lameopts+=(--tv TCOM="$composer")
[ -n "$performer" ] && lameopts+=(--tv TOPE="$performer")
[ -n "$releasecountry" ] \
&& lameopts+=(--tv TXXX="MusicBrainz Album Release Country=$releasecountry")
[ -n "$replaygain_alb" ] \
&& lameopts+=(--tv "TXXX=REPLAYGAIN_ALBUM_GAIN=$replaygain_alb")
[ -n "$replaygain_trk" ] \
&& lameopts+=(--tv "TXXX=REPLAYGAIN_TRACK_GAIN=$replaygain_trk")
[ -n "$disc" ] && lameopts+=(--tv TPOS="$disc")
if (( ${destinationnoresample[$destination]:-0} == 1 ))
then
@ -53,11 +47,12 @@ encodeFile::mp3() {
esac
fi
fi
lameopts+=("$tempdir/$tmpfile.wav" "${destinationpath[$destination]}/$destdir/$destfile.mp3")
lameopts+=("$tempdir/$tmpfile.wav" "$destdir/$destfile.mp3")
encodetaskid=$(
Insert tasks <<-EOInsert
key ${fileid}lame$destination
requires ${soxtaskid:-$decodetaskid}
required ${soxtaskid:-$decodetaskid}
fileid $destfileid
filename $destdir/$destfile.mp3
$(
@ -71,6 +66,7 @@ encodeFile::mp3() {
echo "cmd_arg$key $cleanedopts"
done
)
cleanup $tempdir/$tmpfile.wav
source_file $fileid
status 0
rename_pattern ${destinationrenamepath[$destination]}/${destinationrename[$destination]}
@ -78,12 +74,6 @@ encodeFile::mp3() {
ascii ${destinationascii["$destination"]}
EOInsert
)
parent_required=$(
Select tasks required_by \
<<<"id = ${soxtaskid:-$decodetaskid}"
)
Update tasks required_by $((++parent_required)) \
<<<"id = ${soxtaskid:-$decodetaskid}"
progressSpin
soxtaskid=''
}

View File

@ -1,4 +1,4 @@
#!/usr/bin/env bash
#!/bin/bash
encodeFile::opus() {
opusencopts=(${ionice}opusenc --quiet)
opusencopts+=(--bitrate ${destinationquality[$destination]})
@ -11,23 +11,16 @@ encodeFile::opus() {
[ -n "$disc" ] && opusencopts+=(--comment "DISCNUMBER=$disc")
[ -n "$genre" ] && opusencopts+=(--comment "GENRE=$genre")
[ -n "$performer" ] && opusencopts+=(--comment "PERFORMER=$performer")
[ -n "$releasecountry" ] \
&& opusencopts+=(--comment "RELEASECOUNTRY=$releasecountry")
[ -n "$replaygain_alb" ] \
&& opusencopts+=(--comment) \
&& opusencopts+=("REPLAYGAIN_ALBUM_GAIN=$replaygain_alb")
[ -n "$replaygain_trk" ] \
&& opusencopts+=(--comment) \
&& opusencopts+=("REPLAYGAIN_TRACK_GAIN=$replaygain_trk")
[ -n "$title" ] && opusencopts+=(--title "$title")
[ -n "$track" ] && opusencopts+=(--comment "TRACKNUMBER=${track%/*}")
[ -n "${track#*/}" ] && opusencopts+=(--comment "TRACKTOTAL=${track#*/}")
[ -n "$year" ] && opusencopts+=(--comment "DATE=$year")
opusencopts+=("$tempdir/$tmpfile".wav "${destinationpath[$destination]}/$destdir/$destfile.opus")
opusencopts+=("$tempdir/$tmpfile".wav "$destdir/$destfile.opus")
encodetaskid=$(
Insert tasks <<-EOInsert
key ${fileid}opusenc$destination
requires ${soxtaskid:-$decodetaskid}
required ${soxtaskid:-$decodetaskid}
fileid $destfileid
filename $destdir/$destfile.opus
$(
@ -41,6 +34,7 @@ encodeFile::opus() {
echo "cmd_arg$key $cleanedopts"
done
)
cleanup $tempdir/$tmpfile.wav
source_file $fileid
status 0
rename_pattern ${destinationrenamepath[$destination]}/${destinationrename[$destination]}
@ -48,12 +42,6 @@ encodeFile::opus() {
ascii ${destinationascii["$destination"]}
EOInsert
)
parent_required=$(
Select tasks required_by \
<<<"id = ${soxtaskid:-$decodetaskid}"
)
Update tasks required_by $((++parent_required)) \
<<<"id = ${soxtaskid:-$decodetaskid}"
progressSpin
soxtaskid=''
}

View File

@ -1,4 +1,4 @@
#!/usr/bin/env bash
#!/bin/bash
encodeFile::vorbis() {
oggencopts=(${ionice}oggenc -Q -q ${destinationquality[$destination]})
[ -n "$albumartist" ] && oggencopts+=(-c "ALBUMARTIST=$albumartist")
@ -8,20 +8,15 @@ encodeFile::vorbis() {
[ -n "$disc" ] && oggencopts+=(-c "DISCNUMBER=$disc")
[ -n "$genre" ] && oggencopts+=(-G "$genre")
[ -n "$performer" ] && oggencopts+=(-c "PERFORMER=$performer")
[ -n "$releasecountry" ] \
&& oggencopts+=(-c "RELEASECOUNTRY=$releasecountry")
[ -n "$replaygain_alb" ] \
&& oggencopts+=(-c "REPLAYGAIN_ALBUM_GAIN=$replaygain_alb")
[ -n "$replaygain_trk" ] \
&& oggencopts+=(-c "REPLAYGAIN_TRACK_GAIN=$replaygain_trk")
[ -n "$title" ] && oggencopts+=(-t "$title")
[ -n "$track" ] && oggencopts+=(-N "$track")
[ -n "$year" ] && oggencopts+=(-d "$year")
oggencopts+=(-o "${destinationpath[$destination]}/$destdir/$destfile.ogg" "$tempdir/$tmpfile.wav")
oggencopts+=(-o "$destdir/$destfile.ogg" "$tempdir/$tmpfile.wav")
encodetaskid=$(
Insert tasks <<-EOInsert
key ${fileid}oggenc$destination
requires ${soxtaskid:-$decodetaskid}
required ${soxtaskid:-$decodetaskid}
fileid $destfileid
filename $destdir/$destfile.ogg
$(
@ -35,6 +30,7 @@ encodeFile::vorbis() {
echo "cmd_arg$key ${oggencopts[key]}"
done
)
cleanup $tempdir/$tmpfile.wav
source_file $fileid
status 0
rename_pattern ${destinationrenamepath[$destination]}/${destinationrename[$destination]}
@ -42,12 +38,6 @@ encodeFile::vorbis() {
ascii ${destinationascii["$destination"]}
EOInsert
)
parent_required=$(
Select tasks required_by \
<<<"id = ${soxtaskid:-$decodetaskid}"
)
Update tasks required_by $((++parent_required)) \
<<<"id = ${soxtaskid:-$decodetaskid}"
progressSpin
soxtaskid=''
}

View File

@ -10,10 +10,6 @@ getDestDir() {
[[ ${destinationrenamepath[$destination]} == \
*?([^[])%\{albumartist\}?([^\]])* ]] \
&& [ -n "$albumartist" ]
) || (
[[ ${destinationrenamepath[$destination]} == \
*?([^[])%\{releasecountry\}?([^\]])* ]] \
&& [ -n "$releasecountry" ]
) || (
[[ ${destinationrenamepath[$destination]} == \
*?([^[])%\{artist\}?([^\]])* ]] \
@ -41,15 +37,13 @@ getDestDir() {
)
)
then
destdir=""
destdir="${destinationpath[$destination]}/"
if (( ${destinationascii["$destination"]} ))
then
echo "$album" >&${toascii[1]}
read -r -u${toascii[0]} album
echo "$albumartist" >&${toascii[1]}
read -r -u${toascii[0]} albumartist
echo "$releasecountry" >&${toascii[1]}
read -r -u${toascii[0]} releasecountry
echo "$artist" >&${toascii[1]}
read -r -u${toascii[0]} artist
echo "$genre" >&${toascii[1]}
@ -67,8 +61,6 @@ getDestDir() {
destdir+="${destinationrenamepath[$destination]//?(\[)%\{album\}?(\])/$replace}"
replace=$(sanitizeFile "$albumartist" dir)
destdir="${destdir//?(\[)%\{albumartist\}?(\])/$replace}"
replace=$(sanitizeFile "$releasecountry" dir)
destdir="${destdir//?(\[)%\{releasecountry\}?(\])/$releasecountry}"
replace=$(sanitizeFile "$artist" dir)
destdir="${destdir//?(\[)%\{artist\}?(\])/$replace}"
replace=$(sanitizeFile "$genre" dir)
@ -83,25 +75,24 @@ getDestDir() {
replace=$(sanitizeFile "$disc" dir)
destdir="${destdir//?(\[)%\{disc\}?(\])/$replace}"
else
destdir=$(sanitizeFile "${filename%%/*}" dir)
destdir="${destinationpath[$destination]}/"
destdir+=$(sanitizeFile "${filename%%/*}" dir)
part=${filename#*/}
while [[ $part =~ / ]]
do
thispart="${part%%/*}"
thispart=${thispart//$'\n'/::AtOM:NewLine:SQL:Inline::}
if (( ${destinationascii["$destination"]} ))
then
echo "$thispart" >&${toascii[1]}
read -r -u${toascii[0]} thispart
fi
sanitized="$(sanitizeFile "$thispart" dir)"
destdir+="${sanitized:+/}$sanitized"
destdir+="/$(sanitizeFile "$thispart" dir)"
part=${part#*/}
done
fi
if ! [ -d "${destinationpath[$destination]}/$destdir" ]
if ! [ -d "$destdir" ]
then
mkdir -p "${destinationpath[$destination]}/${destdir//::AtOM:NewLine:SQL:Inline::/$'\n'}"
mkdir -p "$destdir"
fi
destdir="${destdir//+(\/)//}"
}

View File

@ -10,10 +10,6 @@ getDestFile() {
[[ ${destinationrename[$destination]} == \
*?([^[])%\{albumartist\}?([^\]])* ]] \
&& [ -n "$albumartist" ]
) || (
[[ ${destinationrenamepath[$destination]} == \
*?([^[])%\{releasecountry\}?([^\]])* ]] \
&& [ -n "$releasecountry" ]
) || (
[[ ${destinationrename[$destination]} == \
*?([^[])%\{artist\}?([^\]])* ]] \
@ -43,7 +39,6 @@ getDestFile() {
then
destfile="${destinationrename[$destination]//?(\[)%\{album\}?(\])/$album}"
destfile="${destfile//?(\[)%\{albumartist\}?(\])/$albumartist}"
destfile="${destfile//?(\[)%\{releasecountry\}?(\])/$releasecountry}"
destfile="${destfile//?(\[)%\{artist\}?(\])/$artist}"
destfile="${destfile//?(\[)%\{genre\}?(\])/$genre}"
destfile="${destfile//?(\[)%\{title\}?(\])/$title}"
@ -55,11 +50,10 @@ getDestFile() {
destfile="${filename##*/}"
destfile="${destfile%.*}"
fi
destfile=$(sanitizeFile "$destfile")
destfile=${destfile//$'\n'/::AtOM:NewLine:SQL:Inline::}
if (( ${destinationascii["$destination"]} ))
then
echo "$destfile" >&${toascii[1]}
read -r -u${toascii[0]} destfile
fi
destfile=$(sanitizeFile "$destfile")
}

View File

@ -1,6 +1,6 @@
#!/bin/bash
getFiles() {
scantime=$EPOCHSECONDS
scantime=$(date +%s)
for prune_expression in "${skippeddirectories[@]}"
do
prunes+=( -path "$sourcepath$prune_expression" -prune -o )
@ -8,19 +8,12 @@ getFiles() {
(( cron )) || echo -n "Scanning $sourcepath... "
# We probably have thousands of files, don't waste time on disk writes
echo 'BEGIN TRANSACTION;' >&3
while read -d $'\0' time size filename
while read time size filename
do
if (( skip_us_timestamp ))
then
compare_time=${time%.*}.%
else
compare_time=$time
fi
if ! Select source_files id >/dev/null <<-EOWhere
filename = ${filename//$'\n'/::AtOM:NewLine:SQL:Inline::}
filename = $filename
mime_type > 0
last_change LIKE $compare_time
size = $size
last_change = $time
EOWhere
then
mimetype=$(file -b --mime-type "$sourcepath/$filename")
@ -47,7 +40,7 @@ getFiles() {
mime_type $mimetypeid \
>/dev/null \
<<-EOWhere
filename ${filename//$'\n'/::AtOM:NewLine:SQL:Inline::}
filename $filename
EOWhere
(( ++new ))
if (( new % 1000 == 0 ))
@ -58,19 +51,15 @@ getFiles() {
fi
else
Update source_files last_seen $scantime <<-EOWhere
filename = ${filename//$'\n'/::AtOM:NewLine:SQL:Inline::}
filename = $filename
EOWhere
fi
progressSpin
done < <(
find "$sourcepath" "${prunes[@]}" -type f -not -name '.*' -printf "%T@ %s %P\0"
find "$sourcepath" "${prunes[@]}" -type f -not -name '.*' -printf "%T@ %s %P\n"
)
echo 'COMMIT;' >&3
(( cron )) || echo -n $'\r'
if (( count ))
then
echo "$count files found${new:+, $new new or changed}." \
$'\033[K'
fi
echo "${count:-0} files found, ${new:=0} new or changed."$'\033[K'
unset count
}

View File

@ -251,7 +251,6 @@ setupDestination() {
%{artist},
%{disc},
%{genre},
%{releasecountry},
%{title},
%{track},
%{year}.
@ -388,16 +387,16 @@ setupDestination() {
copiedmimes+=("${destinationcopymime["$destination"]%%|*}")
destinationcopymime["$destination"]="${destinationcopymime["$destination"]#*|}"
done
[ -n "${destinationcopymime["$destination"]}" ] \
[ -n "${destinationcopymime["$destination"]}" ] \
&& copiedmimes+=("${destinationcopymime["$destination"]}")
count=${#copiedmimes[@]}
unset destinationcopymime["$destination"]
for (( i=0 ; 1 ; i++ ))
do
read \
-e \
read \
-e \
${copiedmimes[i]+-i"${copiedmimes[i]}"} \
-p 'Copy mime-type: ' \
-p 'Copy mime-type: ' \
value
if [ -n "$value" ]
then
@ -410,44 +409,6 @@ setupDestination() {
fi
done
unset copiedmimes
cat <<-EODesc
Copy extensions (extension, string):
Files with extension <extension> will be copied as-is to the
destination. E.g. .jpg will copy covers and other images to the
destination.
This prompt will loop until an empty string is encountered.
EODesc
while [[ ${destinationcopyext["$destination"]} =~ \| ]]
do
copiedexts+=("${destinationcopyext["$destination"]%%|*}")
destinationcopyext["$destination"]="${destinationcopyext["$destination"]#*|}"
done
[ -n "${destinationcopyext["$destination"]}" ] \
&& copiedexts+=("${destinationcopyext["$destination"]}")
count=${#copiedexts[@]}
unset destinationcopyext["$destination"]
for (( i=0 ; 1 ; i++ ))
do
read \
-e \
${copiedexts[i]+-i"${copiedexts[i]}"} \
-p 'Copy extension: ' \
value
if [ -n "$value" ]
then
destinationcopyext[$destination]="${destinationcopyext[$destination]:+${destinationcopyext[$destination]}|}$value"
elif (( i < count ))
then
continue
else
break
fi
done
unset copiedexts
cat <<-EODesc
Channels (integer):

View File

@ -24,7 +24,6 @@ Completion is available for prompts asking for a paths or filenames.
destinationchannels \
destinationfat32compat \
destinationcopymime \
destinationcopyext \
destinationformat \
destinationfrequency \
destinationid \
@ -41,7 +40,6 @@ Completion is available for prompts asking for a paths or filenames.
destinationchannels \
destinationfat32compat \
destinationcopymime \
destinationcopyext \
destinationformat \
destinationfrequency \
destinationid \
@ -76,7 +74,7 @@ Completion is available for prompts asking for a paths or filenames.
esac
;;
esac
read -p'Run index and conversion now? [Y/n] ' do_run
read -p'Run now? [Y/n] ' do_run
case $do_run in
n) exit ;;
*) ;;

55
lib/tags/ffmpeg Normal file
View File

@ -0,0 +1,55 @@
#!/bin/bash
getInfosffmpeg_version='ffmpeg-6'
tagreaders+=( "$getInfosffmpeg_version" )
getInfos::ffmpeg() {
tagreader="$getInfosffmpeg_version"
local allinfos=$(
ffprobe -show_streams \
-i "$sourcepath/$filename" 2>&1 \
|sed '
/^Input/,/.* Audio: /{s/ *: */=/}
s/^[[:space:]]*//
s/\0//g'
)
local metadata=$(
echo -e "$allinfos" \
|sed -n '/Metadata=/,/\[STREAM\]/p'
)
local fmt_infos=$(
echo -e "$allinfos" \
|sed -n \
'/codec_type=audio/,/\[STREAM\]/{
/^\(sample_fmt\|sample_rate\|bit_rate\|channels\)=/{
p
}
}'
)
local infos="$metadata"
albumartist=$(gettag album_artist)
album=$(gettag album)
artist=$(gettag artist)
composer=$(gettag composer)
disc=$(gettag disc)
genre=$(gettag genre)
performer=$(gettag TOPE)
title=$(gettag title)
tracknum=$(gettag track)
year=$(gettag date)
expr='^[0-9]*$'
if [ -n "$genre" ] && [[ $genre =~ $expr ]]
then
genre="${id3genres[$genre]}"
fi
infos="$fmt_infos"
channels=$(gettag channels)
rate=$(gettag 'sample_rate')
bitrate=$(gettag 'bit_rate')
bitdepth=$(gettag 'sample_fmt')
bitdepth=${bitdepth//[A-z]/}
if [[ $bitrate == N/A ]]
then
unset bitrate
else
bitrate=$((bitrate / 1000))
fi
}

48
lib/tags/flac Normal file
View File

@ -0,0 +1,48 @@
#!/bin/bash
getInfosFLAC_version='FLAC-3'
tagreaders+=( "$getInfosFLAC_version" )
getInfos::FLAC() {
tagreader="$getInfosFLAC_version"
infos=$(
metaflac \
--show-tag=ALBUM \
--show-tag=ALBUMARTIST \
--show-tag=ARTIST \
--show-tag=COMPOSER \
--show-tag=DATE \
--show-tag=DISCNUMBER \
--show-tag=GENRE \
--show-tag=PERFORMER \
--show-tag=TITLE \
--show-tag=TRACKNUMBER \
--show-tag=TRACKTOTAL \
"$sourcepath/$filename"
)
albumartist=$(gettag albumartist)
album=$(gettag album)
artist=$(gettag artist)
composer=$(gettag composer)
disc=$(gettag discnumber)
genre=$(gettag genre)
performer=$(gettag performer)
title=$(gettag title)
tracknum=$(gettag tracknumber)
tracktotal=$(gettag tracktotal)
year=$(gettag date)
if [ -n "$tracknum" -a -n "$tracktotal" ]
then
tracknum="$tracknum/$tracktotal"
fi
year=$(gettag DATE)
{
read rate
read channels
read bitdepth
} < <(
metaflac \
--show-sample-rate \
--show-channels \
--show-bps \
"$sourcepath/$filename"
)
}

View File

@ -1,54 +0,0 @@
#!/usr/bin/env bash
getInfosFLAC_version='FLAC-5'
tagreaders+=( "$getInfosFLAC_version" )
getInfos::FLAC() {
local \
infos \
tagreader="$getInfosFLAC_version"
infos=$(
metaflac \
--show-sample-rate \
--show-channels \
--show-bps \
--show-tag=ALBUM \
--show-tag=ALBUMARTIST \
--show-tag=ARTIST \
--show-tag=COMPOSER \
--show-tag=DATE \
--show-tag=DISCNUMBER \
--show-tag=GENRE \
--show-tag=PERFORMER \
--show-tag=RELEASECOUNTRY \
--show-tag=REPLAYGAIN_ALBUM_GAIN \
--show-tag=REPLAYGAIN_TRACK_GAIN \
--show-tag=TITLE \
--show-tag=TRACKNUMBER \
--show-tag=TRACKTOTAL \
"$sourcepath/$filename"
)
albumartist=$(gettag albumartist)
album=$(gettag album)
artist=$(gettag artist)
composer=$(gettag composer)
disc=$(gettag discnumber)
genre=$(gettag genre)
performer=$(gettag performer)
releasecountry=$(gettag releasecountry)
replaygain_alb=$(gettag replaygain_album_gain)
replaygain_trk=$(gettag replaygain_track_gain)
title=$(gettag title)
tracknum=$(gettag tracknumber)
tracktotal=$(gettag tracktotal)
year=$(gettag date)
if [ -n "$tracknum" -a -n "$tracktotal" ]
then
tracknum="$tracknum/$tracktotal"
fi
year=$(gettag DATE)
{
read rate
read channels
read depth
} <<<"$infos"
}

View File

@ -1,75 +0,0 @@
#!/usr/bin/env bash
getInfosffmpeg_version='ffmpeg-9'
tagreaders+=( "$getInfosffmpeg_version" )
getInfos::ffmpeg() {
tagreader="$getInfosffmpeg_version"
local \
infos \
infos=$(
ffprobe -v error \
-show_entries " \
format_tags= \
album_artist, \
album, \
artist, \
composer, \
disc, \
genre, \
TOPE, \
releasecountry, \
'MusicBrainz Album Release Country',\
title, \
track, \
date, \
replaygain_track_gain, \
replaygain_album_gain \
:stream= \
bit_rate, \
channels, \
sample_rate, \
" \
-of default=noprint_wrappers=1 \
-i "$sourcepath/$filename" \
| egrep -v '=N/A$'
)
albumartist=$(gettag TAG:album_artist)
album=$(gettag TAG:album)
artist=$(gettag TAG:artist)
composer=$(gettag TAG:composer)
disc=$(gettag TAG:disc)
genre=$(gettag TAG:genre)
performer=$(gettag TAG:TOPE)
releasecountry=$(gettag TAG:releasecountry)
[[ -z "$releasecountry" ]] \
&& releasecountry=$(gettag "TAG:MusicBrainz Album Release Country")
replaygain_alb=$(gettag TAG:replaygain_album_gain)
replaygain_trk=$(gettag TAG:replaygain_track_gain)
title=$(gettag TAG:title)
tracknum=$(gettag TAG:track)
year=$(gettag TAG:date)
expr='^[0-9]*$'
if [ -n "$genre" ] && [[ $genre =~ $expr ]]
then
genre="${id3genres[$genre]}"
fi
channels=$(gettag channels)
rate=$(gettag 'sample_rate')
case $rate in
96) rate=96000;;
48) rate=48000;;
441) rate=44100;;
32) rate=32000;;
24) rate=24000;;
225) rate=22500;;
esac
bitrate=$(gettag 'bit_rate')
depth=$(gettag 'sample_fmt')
depth=${depth//[A-z]/}
if [[ $bitrate == N/A ]]
then
unset bitrate
else
bitrate=$((bitrate / 1000))
fi
}

View File

@ -1,7 +1,7 @@
#!/bin/bash
getTags_version='unknown-4'
getTags() {
local type
unset type
case "$mimetype" in
audio/mpeg)
type=ffmpeg
@ -11,15 +11,11 @@ getTags() {
type=Opus
(( disableopusinfo )) && unset type
;;
'audio/ogg opus')
type=Opus
(( disableopusinfo )) && unset type
;;
application/ogg*)
type=soxi
(( disablesoxi )) && unset type
;;
audio/ogg*)
audio/ogg)
type=soxi
(( disablesoxi )) && unset type
;;

View File

@ -1,11 +1,8 @@
#!/usr/bin/env bash
getInfosOpus_version='Opus-4'
#!/bin/bash
getInfosOpus_version='Opus-2'
tagreaders+=( "$getInfosOpus_version" )
getInfos::Opus() {
tagreader="$getInfosOpus_version"
local \
infos \
infos=$(
opusinfo "$sourcepath/$filename" \
| sed 's/\t//'
@ -17,9 +14,6 @@ getInfos::Opus() {
disc=$(gettag discnumber)
genre=$(gettag genre)
performer=$(gettag performer)
releasecountry=$(gettag releasecountry)
replaygain_alb=$(gettag replaygain_album_gain)
replaygain_trk=$(gettag replaygain_track_gain)
title=$(gettag title)
tracknum=$(gettag tracknumber)
tracktotal=$(gettag tracktotal)
@ -30,9 +24,7 @@ getInfos::Opus() {
year=$(gettag date)
infos="${infos//: /=}"
rate=$(gettag 'original sample rate'|head -n1)
rate=${rate% Hz}
channels=$(gettag channels|head -n1)
bitrate=$(gettag 'average bitrate')
bitrate=${bitrate%% kbit*}
bitrate=${bitrate%%.*}
}

View File

@ -1,14 +1,10 @@
#!/usr/bin/env bash
getInfosSoxi_version='soxi-3'
#!/bin/bash
getInfosSoxi_version='soxi-1'
tagreaders+=( "$getInfosSoxi_version" )
getInfos::soxi() {
tagreader="$getInfosSoxi_version"
local \
infos \
infos=$(
soxi "$sourcepath/$filename" \
| grep -v METADATA_BLOCK_PICTURE
soxi "$sourcepath/$filename"
)
albumartist=$(gettag albumartist)
album=$(gettag album)
@ -17,9 +13,6 @@ getInfos::soxi() {
disc=$(gettag discnumber)
genre=$(gettag genre)
performer=$(gettag performer)
releasecountry=$(gettag releasecountry)
replaygain_alb=$(gettag replaygain_album_gain)
replaygain_trk=$(gettag replaygain_track_gain)
title=$(gettag title)
tracknum=$(gettag tracknumber)
tracktotal=$(gettag tracktotal)
@ -34,6 +27,6 @@ getInfos::soxi() {
bitrate=$(gettag 'bit rate')
bitrate=${bitrate%k}
bitrate=${bitrate%%.*}
depth=$(gettag precision)
depth=${depth%-bit}
bitdepth=$(gettag precision)
bitdepth=${bitdepth%-bit}
}

6
lib/tags/tryAPE Normal file
View File

@ -0,0 +1,6 @@
#!/bin/bash
tryAPE() {
grep -q 'APETAGEX' \
"$sourcepath/$filename" \
&& type=APE
}

View File

@ -1,7 +1,5 @@
#!/usr/bin/env bash
#!/bin/bash
updateTags() {
local reader \
for reader in "${tagreaders[@]}"
do
tagreaderclause+="${tagreaderclause:+ AND }NOT tags.tagreader = \"$reader\""
@ -20,9 +18,6 @@ updateTags() {
tags.disc,
tags.genre,
tags.performer,
tags.releasecountry,
tags.replaygain_alb,
tags.replaygain_trk,
tags.title,
tags.track,
tags.year,
@ -52,12 +47,12 @@ echo '
;
SELECT "AtOM:NoMoreFiles";' >&3
read -u4 -r -d $'\0' line
read -u4 line
while ! [[ $line = AtOM:NoMoreFiles ]]
do
tagfiles+=("$line")
(( filecount++ ))
read -u4 -r -d $'\0' line
read -u4 line
done
echo 'BEGIN TRANSACTION;' >&3
for line in "${tagfiles[@]}"
@ -86,12 +81,6 @@ echo '
rest=${rest#*::AtOM:SQL:Sep::}
oldperformer=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
oldreleasecountry=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
oldreplaygain_alb=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
oldreplaygain_trk=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
oldtitle=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
oldtrack=${rest%%::AtOM:SQL:Sep::*}
@ -105,7 +94,7 @@ echo '
oldbitrate=${rest%%::AtOM:SQL:Sep::*}
((++count))
(( cron )) || echo -en "\rTags: $((count*100/filecount))%"
if (( count % 100 == 0 ))
if (( count % 1000 == 0 ))
then
echo 'COMMIT;BEGIN TRANSACTION;' >&3
(( debug )) \
@ -117,13 +106,10 @@ echo '
[[ $oldalbumartist != "$albumartist" ]]&&uaa=1
[[ $oldartist != "$artist" ]]&& uar=1
[[ $oldcomposer != "$composer" ]]&& uco=1
[[ $olddepth != "$depth" ]]&& ude=1
[[ $olddepth != "$bitdepth" ]]&& ude=1
[[ $olddisc != "$disc" ]]&& udi=1
[[ $oldgenre != "$genre" ]]&& uge=1
[[ $oldperformer != "$performer" ]]&& upe=1
[[ $oldreleasecountry != "$releasecountry" ]]&& urc=1
[[ $oldreplaygain_alb != "$replaygain_alb" ]]&& urpa=1
[[ $oldreplaygain_trk != "$replaygain_trk" ]]&& urpt=1
[[ $oldtitle != "$title" ]]&& uti=1
[[ $oldtrack != "$tracknum" ]]&& utr=1
[[ $oldyear != "$year" ]]&& uye=1
@ -135,13 +121,10 @@ echo '
${uaa:+albumartist "${albumartist:+::AtOM:FT::}${albumartist:-NULL}"}\
${uar:+artist "${artist:+::AtOM:FT::}${artist:-NULL}"}\
${uco:+composer "${composer:+::AtOM:FT::}${composer:-NULL}"}\
${ude:+depth "${depth:-NULL}"} \
${ude:+depth "${bitdepth:-NULL}"} \
${udi:+disc "${disc:-NULL}"} \
${uge:+genre "${genre:-NULL}"} \
${upe:+performer "${performer:+::AtOM:FT::}${performer:-NULL}"}\
${urc:+releasecountry "${releasecountry:+::AtOM:FT::}${releasecountry:-NULL}"}\
${urpa:+replaygain_alb "${replaygain_alb:-NULL}"}\
${urpt:+replaygain_trk "${replaygain_trk:-NULL}"}\
${uti:+title "${title:+::AtOM:FT::}${title:-NULL}"}\
${utr:+track "${track:+::AtOM:FT::}${tracknum:-NULL}"}\
${uye:+year "${year:-NULL}"} \
@ -150,64 +133,41 @@ echo '
${uch:+channels "${channels:-NULL}"} \
${ubi:+bitrate "${bitrate:-NULL}"} \
tagreader "$tagreader" \
>/dev/null <<<"source_file = $sourcefileid"
unset genre \
albumartist \
year \
album \
disc \
artist \
tracknum \
title \
composer \
performer \
releasecountry \
replaygain_alb \
replaygain_trk \
rate \
depth \
bitrate \
channels \
oldgenre \
oldalbumartist \
oldyear \
oldalbum \
olddisc \
oldartist \
oldtracknum \
oldtitle \
oldcomposer \
oldperformer \
oldreleasecountry \
oldreplaygain_alb \
oldreplaygain_trk \
oldrate \
olddepth \
oldbitrate \
oldchannels \
ual \
uaa \
uar \
uco \
ude \
udi \
uge \
upe \
urc \
urpa \
urpt \
uti \
utr \
uye \
ura \
uch \
ubi
>/dev/null <<<"source_file = $sourcefileid"
unset genre \
albumartist \
year \
album \
disc \
artist \
tracknum \
title \
composer \
performer \
rate \
bitdepth \
bitrate \
channels \
ual \
uaa \
uar \
uco \
ude \
udi \
uge \
upe \
uti \
utr \
uye \
ura \
uch \
ubi
fi
done
echo 'COMMIT;' >&3
(( cron )) || echo -n $'\r'
(( count )) && echo -n "Read tags from $count files."
(( cron )) || echo -n $'\033[K'
(( count )) && echo
echo -n "Read tags from ${count:-0} files."
(( cron )) || echo -ne "\033[K"
echo
unset count tagfiles
}

View File

@ -4,17 +4,20 @@ gettaskinfos() {
SELECT
id,
source_file,
required,
cleanup,
fileid,
filename
FROM tasks
WHERE id='$1';
' >&3
read -u4 -r -d $'\0' line
read -u4 line
taskid=${line%%::AtOM:SQL:Sep::*}
rest="${line#*::AtOM:SQL:Sep::}::AtOM:SQL:Sep::"
sourcefileid=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
required=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
cleanup=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
destfileid=${rest%%::AtOM:SQL:Sep::*}

View File

@ -1,166 +0,0 @@
#!/usr/bin/env bash
sanityCheck() {
if [ ! -d "$tempdir" ] && ! mkdir -p "$tempdir"
then
echo "[FATAL] Could not create temp directory $tempdir" >&2
(( sanityfail++ ))
fi
if [ ! -f "$database" ] && [ ! -d "${database%/*}" ] \
&& ! mkdir -p "${database%/*}"
then
echo "[FATAL] Directory holding database file does not exist"\
"and could not be created" >&2
(( sanityfail++ ))
fi
if [ ! -d "$sourcepath" ]
then
echo "[FATAL] Source path $sourcepath does not exist or is"\
"not a directory" >&2
(( sanityfail++ ))
fi
if ! which sed >/dev/null
then
echo "[FATAL] Required tool sed is not installed or not in PATH
I never thought this would actually hit someone..." >&2
(( sanityfail++ ))
fi
if ! which sox >/dev/null
then
echo "[FATAL] Required tool sox is not installed or not in"\
"PATH" >&2
(( sanityfail++ ))
fi
if ! which ogginfo >/dev/null
then
echo "[WARNING] Tool ogginfo (from vorbis-tools) is not"\
"installed or not in PATH
WebM metadata disabled" >&2
disableogginfo=1
(( sanitywarn++ ))
fi
if ! which soxi >/dev/null
then
echo "[WARNING] Tool soxi (from sox) is not" \
"installed or not in PATH
Vorbis metadata disabled" >&2
disablesoxi=1
(( sanitywarn++ ))
fi
if (( oggencneeded )) && ! which oggenc >/dev/null
then
echo "[WARNING] Tool oggenc (from vorbis-tools) is not" \
"installed or not in PATH
Vorbis targets disabled" >&2
disableoggenc=1
(( sanitywarn++ ))
fi
if ! which opusinfo >/dev/null
then
echo "[WARNING] Tool opusinfo (from opus-tools) is not" \
"installed or not in PATH
Opus metadata disabled" >&2
disableopusinfo=1
(( sanitywarn++ ))
fi
if (( opusencneeded )) && ! which opusenc >/dev/null
then
echo "[WARNING] Tool opusenc (from opus-tools) is not" \
"installed or not in PATH
Opus targets disabled" >&2
disableopusenc=1
(( sanitywarn++ ))
fi
if ! which opusdec >/dev/null
then
echo "[WARNING] Tool opusdec (from opus-tools) is not" \
"installed or not in PATH
Opus support disabled" >&2
disableopusdec=1
(( sanitywarn++ ))
fi
if (( lameneeded )) && ! which lame >/dev/null
then
echo "[WARNING] Tool lame is not installed or not in PATH
MP3 targets disabled" >&2
disablelame=1
(( sanitywarn++ ))
fi
if ! which metaflac >/dev/null
then
echo "[WARNING] Tool metaflac (from FLAC) is not installed"\
"or not in PATH
FLAC metadata disabled" >&2
disableflac=1
(( sanitywarn++ ))
fi
if ! which mpcdec >/dev/null
then
echo "[WARNING] Tool mpcdec (from Musepack) is not" \
"installed or not in PATH
Musepack support disabled" >&2
disablempcdec=1
(( sanitywarn++ ))
fi
if ! which mkvextract >/dev/null
then
echo "[WARNING] Tool mkvextract (from MKVToolNix) is not"\
"installed or not in PATH
WebM metadata disabled
WebM support disabled" >&2
disablemkvextract=1
(( sanitywarn++ ))
fi
if ! which ffprobe >/dev/null
then
echo "[WARNING] Tool ffprobe (from FFmpeg) is not installed"\
"or not in PATH
Video metadata disabled
MPEG metadata disabled
MusePack metadata disabled
Unknown format metadata disabled" >&2
disableffprobe=1
(( sanitywarn++ ))
fi
if ! which ffmpeg >/dev/null
then
echo "[WARNING] Tool ffmpeg is not installed or not in PATH
Video support disabled" >&2
disablevideo=1
(( sanitywarn++ ))
fi
if (( textunidecodeneeded )) && ! perl -MText::Unidecode -e 'exit;' 2>/dev/null
then
echo "[WARNING] Perl module Text::Unidecode is not available
Renaming to ASCII-only disabled" >&2
unset destinationascii
destinationascii=0
textunidecodeneeded=0
(( sanitywarn++ ))
fi
if (( sanityfail ))
then
echo "
Sanity checks raised ${sanitywarn:-0} warnings, $sanityfail failures. Dying now." >&2
exit $ESANITY
elif (( sanitywarn ))
then
echo "
Sanity checks raised $sanitywarn warnings... Hit Control-C to abort." >&2
if ! (( cron ))
then
timeout=$(( sanitywarn * 10 ))
echo -n "Starting in $(printf %3i $timeout)" \
$'seconds...\b\b\b\b\b\b\b\b\b\b\b' >&2
while (( timeout ))
do
echo -n $'\b\b\b'"$(printf %3i $timeout)" >&2
sleep 1
(( timeout-- ))
done
echo -en "\r\033[K"
fi
fi
}
# vim:set ts=8 sw=8:

View File

@ -1,9 +1,5 @@
#!/bin/bash
checkworkers() {
local \
taskid \
parent_required \
parent_task
for key in ${!workers[@]}
do
if ! kill -0 ${workers[key]} 2>/dev/null
@ -18,19 +14,6 @@ checkworkers() {
failedtasks+=($taskid)
(( ++failed ))
fi
parent_task=$(
Select tasks requires \
<<<"id = $taskid"
)
if (( parent_task ))
then
parent_required=$(
Select tasks required_by \
<<<"id = $parent_task"
)
Update tasks required_by $((--parent_required)) \
<<<"id = $parent_task"
fi
unset workertasks[key]
fi
done

View File

@ -1,23 +1,26 @@
#!/bin/bash
cleaner() {
local \
key \
faildepends \
taskid \
count
for key in ${!failedtasks[@]}
do
taskid=${failedtasks[key]}
gettaskinfos $taskid
faildepends=$(
Select tasks required_by <<-EOWhere
id = $taskid
Select tasks 'COUNT(*)' <<-EOWhere
requires = $taskid
EOWhere
)
(( failed+=faildepends ))
(( ran+=faildepends ))
Update tasks status 2 <<<"id = $taskid"
rm -f "$cleanup"
echo "SELECT COUNT(*)
FROM tasks
WHERE ( status = 0 OR status = 1 )
AND required = $taskid;">&3
read -u4 count
if (( count == 0 ))
then
rm -f "$cleanup"
fi
unset failedtasks[key]
done
for key in ${!finishedtasks[@]}
@ -44,28 +47,29 @@ cleaner() {
" FROM tasks" \
" WHERE id=$taskid" \
" )," \
" fat32compat=(" \
" fat32compat=(" \
" SELECT fat32compat" \
" FROM tasks" \
" WHERE id=$taskid" \
" )," \
" ascii=(" \
" SELECT ascii" \
" ascii=(" \
" SELECT ascii" \
" FROM tasks" \
" WHERE id=$taskid" \
" )" \
"WHERE id=$destfileid;" \
>&3
fi
count=$(Select tasks required_by <<<"id = $taskid")
echo "SELECT COUNT(*)
FROM tasks
WHERE ( status = 0 OR status = 1 )
AND required = $taskid;">&3
read -u4 count
if (( count == 0 ))
then
[[ -n "$cleanup" ]] && rm -f "$tempdir/$cleanup"
Delete tasks <<<"id = $taskid"
unset finishedtasks[key]
else
Update tasks status 4 \
<<<"id = $taskid"
rm -f "$cleanup"
fi
Delete tasks <<<"id = $taskid"
unset finishedtasks[key]
done
}

View File

@ -1,145 +1,5 @@
#!/usr/bin/env bash
#!/bin/bash
createworker() {
(( ++active ))
read -u4 -r -d $'\0' line
taskid=${line%%::AtOM:SQL:Sep::*}
rest="${line#*::AtOM:SQL:Sep::}::AtOM:SQL:Sep::"
sourcefileid=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cleanup=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
destfileid=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
destfilename=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
for key in ${!cmd_arg[@]}
do
[ -z "${cmd_arg[key]}" ] && unset cmd_arg[key]
done
workerid=$(getworkerid)
workertasks[workerid]=$taskid
Update tasks status 1 <<<"id = $taskid"
worker $workerid &
workers[$workerid]=$!
worker $1 &
workers[$1]=$!
}

View File

@ -8,233 +8,261 @@ master() {
SELECT COUNT(*)
FROM tasks
WHERE status = 0;
'>&3
read -u4 -r -d $'\0' remaining
SELECT COUNT(*)
FROM tasks
WHERE status = 0
AND requires is NULL;
SELECT
id,
source_file,
required,
cmd_arg0,
cmd_arg1,
cmd_arg2,
cmd_arg3,
cmd_arg4,
cmd_arg5,
cmd_arg6,
cmd_arg7,
cmd_arg8,
cmd_arg9,
cmd_arg10,
cmd_arg11,
cmd_arg12,
cmd_arg13,
cmd_arg14,
cmd_arg15,
cmd_arg16,
cmd_arg17,
cmd_arg18,
cmd_arg19,
cmd_arg20,
cmd_arg21,
cmd_arg22,
cmd_arg23,
cmd_arg24,
cmd_arg25,
cmd_arg26,
cmd_arg27,
cmd_arg28,
cmd_arg29,
cmd_arg30,
cmd_arg31,
cmd_arg32,
cmd_arg33,
cmd_arg34,
cmd_arg35,
cmd_arg36,
cmd_arg37,
cmd_arg38,
cmd_arg39,
cmd_arg40,
cmd_arg41,
cmd_arg42,
cmd_arg43,
cmd_arg44,
cmd_arg45,
cmd_arg46,
cmd_arg47,
cmd_arg48,
cmd_arg49,
cmd_arg50,
cmd_arg51,
cmd_arg52,
cmd_arg53,
cmd_arg54,
cmd_arg55,
cmd_arg56,
cmd_arg57,
cmd_arg58,
cmd_arg59,
cleanup,
fileid,
filename
FROM tasks
WHERE status = 0
AND requires is NULL
ORDER BY source_file
LIMIT 1;
' >&3
read -u4 remaining
read -u4 ready
if (( remaining == 0 ))
then
sleep 0.1
return 0
elif (( active == 0 && ready == 0 ))
then
dumpfile=tasks-$(date +%Y%m%d%H%M%S).csv
cat <<-EOF
$remaining TASKS LEFT, NONE READY!
Something went wrong, dumping tasks table to $dumpfile
EOF
cat >&3 <<-EOSQL
.mode csv
.headers on
.output $dumpfile
SELECT * from tasks;
.mode list
.headers off
.output stdout
EOSQL
closeDatabase
echo "Waiting for children to come back home..."
wait
echo $'\nGood luck!'
exit 1
elif (( ready == 0 ))
then
sleep 0.1
else
(( ++active ))
read -u4 line
taskid=${line%%::AtOM:SQL:Sep::*}
rest="${line#*::AtOM:SQL:Sep::}::AtOM:SQL:Sep::"
sourcefileid=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
required=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cmd_arg+=("${rest%%::AtOM:SQL:Sep::*}")
rest=${rest#*::AtOM:SQL:Sep::}
cleanup=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
destfileid=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
destfilename=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
for key in ${!cmd_arg[@]}
do
[ -z "${cmd_arg[key]}" ] && unset cmd_arg[key]
done
workerid=$(getworkerid)
workertasks[workerid]=$taskid
Update tasks status 1 <<<"id = $taskid"
createworker $workerid
fi
until (( active == concurrency || remaining == 0 ))
do
echo '
SELECT COUNT(*)
FROM tasks
WHERE status = 0
AND requires IN (
SELECT id
FROM tasks
WHERE status = 4
);
SELECT
id,
source_file,
cmd_arg0,
cmd_arg1,
cmd_arg2,
cmd_arg3,
cmd_arg4,
cmd_arg5,
cmd_arg6,
cmd_arg7,
cmd_arg8,
cmd_arg9,
cmd_arg10,
cmd_arg11,
cmd_arg12,
cmd_arg13,
cmd_arg14,
cmd_arg15,
cmd_arg16,
cmd_arg17,
cmd_arg18,
cmd_arg19,
cmd_arg20,
cmd_arg21,
cmd_arg22,
cmd_arg23,
cmd_arg24,
cmd_arg25,
cmd_arg26,
cmd_arg27,
cmd_arg28,
cmd_arg29,
cmd_arg30,
cmd_arg31,
cmd_arg32,
cmd_arg33,
cmd_arg34,
cmd_arg35,
cmd_arg36,
cmd_arg37,
cmd_arg38,
cmd_arg39,
cmd_arg40,
cmd_arg41,
cmd_arg42,
cmd_arg43,
cmd_arg44,
cmd_arg45,
cmd_arg46,
cmd_arg47,
cmd_arg48,
cmd_arg49,
cmd_arg50,
cmd_arg51,
cmd_arg52,
cmd_arg53,
cmd_arg54,
cmd_arg55,
cmd_arg56,
cmd_arg57,
cmd_arg58,
cmd_arg59,
cleanup,
fileid,
filename
FROM tasks
WHERE status = 0
AND requires IN (
SELECT id
FROM tasks
WHERE status = 4
ORDER BY source_file
/* LIMIT 1 */
)
ORDER BY source_file
LIMIT 1;
'>&3
read -u4 -r -d $'\0' ready
if (( ready > 0 ))
then
createworker
continue
fi
echo '
SELECT COUNT(*)
FROM tasks
WHERE status = 0
AND requires is NULL;
SELECT
id,
source_file,
cmd_arg0,
cmd_arg1,
cmd_arg2,
cmd_arg3,
cmd_arg4,
cmd_arg5,
cmd_arg6,
cmd_arg7,
cmd_arg8,
cmd_arg9,
cmd_arg10,
cmd_arg11,
cmd_arg12,
cmd_arg13,
cmd_arg14,
cmd_arg15,
cmd_arg16,
cmd_arg17,
cmd_arg18,
cmd_arg19,
cmd_arg20,
cmd_arg21,
cmd_arg22,
cmd_arg23,
cmd_arg24,
cmd_arg25,
cmd_arg26,
cmd_arg27,
cmd_arg28,
cmd_arg29,
cmd_arg30,
cmd_arg31,
cmd_arg32,
cmd_arg33,
cmd_arg34,
cmd_arg35,
cmd_arg36,
cmd_arg37,
cmd_arg38,
cmd_arg39,
cmd_arg40,
cmd_arg41,
cmd_arg42,
cmd_arg43,
cmd_arg44,
cmd_arg45,
cmd_arg46,
cmd_arg47,
cmd_arg48,
cmd_arg49,
cmd_arg50,
cmd_arg51,
cmd_arg52,
cmd_arg53,
cmd_arg54,
cmd_arg55,
cmd_arg56,
cmd_arg57,
cmd_arg58,
cmd_arg59,
cleanup,
fileid,
filename
FROM tasks
WHERE status = 0
AND requires is NULL
ORDER BY source_file
LIMIT 1;
' >&3
read -u4 -r -d $'\0' ready
if (( active == 0 && ready == 0 ))
then
dumpfile="$tempdir/tasks-$(date -Iseconds).csv"
cat <<-EOF
$remaining TASKS LEFT, NONE READY!
Something went wrong, dumping tasks table to $dumpfile
EOF
cat >&3 <<-EOSQL
.mode csv
.headers on
.output $dumpfile
SELECT * from tasks;
.mode list
.headers off
.output stdout
COMMIT;
EOSQL
closeDatabase
echo "Waiting for children to come back home..."
wait
echo $'\nGood luck!'
exit $ETASKLEFT
elif (( ready == 0 ))
then
sleep 0.1
break
else
createworker
fi
echo '
SELECT COUNT(*)
FROM tasks
WHERE status = 0;
'>&3
read -u4 -r -d $'\0' remaining
done
fi
}

View File

@ -1,35 +0,0 @@
#!/usr/bin/env bash
# https://specifications.freedesktop.org/basedir-spec/latest/
xdgUpdateData() {
local -r programname=AtOM
local -r my_data_home="${XDG_DATA_HOME:-$HOME/.local/share}/$programname"
local -r new_db_path="$my_data_home/atom.db"
[[ -d "$my_data_home" ]] || mkdir -p "$my_data_home"
mv "$1" "$new_db_path"
echo "$new_db_path"
}
xdgUpdateRuntime() {
local -r programname=AtOM \
my_runtime_dir="${XDG_RUNTIME_DIR:-/tmp}" \
echo "$my_runtime_dir/$programname"
}
xdgMigrate() {
local -r programname=AtOM
local -r my_config_home="${XDG_CONFIG_HOME:-$HOME/.config}/$programname"
local new_database
cffile="$HOME/.atom/atom.cfg"
getConfig
new_database=$(xdgUpdateData "$database")
database="$new_database"
tempdir=$(xdgUpdateRuntime)
cffile="$my_config_home/atom.cfg"
[[ -d "$my_config_home" ]] || mkdir -p "$my_config_home"
writeConfig >"$cffile"
}

View File

@ -1,24 +0,0 @@
#!/usr/bin/env bash
## Define exit codes
# General config errors [10-19]
EDBVERSION=10
ELOAD=11
EINTERVAL=12
ELOAD=13
EIONICE=14
ENOCFG=19
# Source cofig errors [20-29]
# Destination config errors [30-49]
EFORMAT=30
ECHANNEL=31
EQUALITY=32
EMAXBPS=33
EINVDEST=34
EFMTINVPARM=49
# Tasks
ETASKLEFT=50
# Sanity
ESANITY=60
# Invalid arguments
EINVARG=127

View File

@ -58,9 +58,6 @@ CREATE TABLE IF NOT EXISTS tags (
title TEXT,
composer TEXT,
performer TEXT,
releasecountry TEXT,
replaygain_alb TEXT,
replaygain_trk TEXT,
depth INTEGER,
rate INTEGER,
channels INTEGER,
@ -132,13 +129,10 @@ CREATE TRIGGER IF NOT EXISTS force_destination_update_on_tag_update
title,
composer,
performer,
releasecountry,
replaygain_alb,
replaygain_trk,
rate,
channels,
bitrate,
depth
bitdepth
ON tags
BEGIN
UPDATE destination_files SET last_change=0

View File

@ -1,26 +1,5 @@
#!/usr/bin/env bash
cat <<-EOWarn
This script is unmaintained and provided as-is. It may or may not work.
Use at your own risk!
EOWarn
read -p "Press Enter to continue or Ctrl-C to abort"
declare -r \
DOCDIR=%DOCDIR% \
LIBDIR=%LIBDIR% \
SHAREDIR=%SHAREDIR%
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
## Define exit codes
source "$SHAREDIR"/errorcodes
# config structures
declare -A \
destinationchannels \
@ -43,6 +22,17 @@ declare -A \
exit $EBASHVERS
}
declare -r \
DOCDIR=./doc \
LIBDIR=./lib \
SHAREDIR=./share
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
cffile="$HOME/.atom/atom.cfg"
LC_ALL=C
shopt -s extglob
@ -52,15 +42,6 @@ do
source "$function"
done
if ! [[ -f "${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg" ]] \
&& [[ -f "$HOME/.atom/atom.cfg" ]]
then
echo "Configuration found in legacy location $HOME/.atom/atom.cfg."\
"Migrating configuration and data to XDG standard"
xdgMigrate
fi
cffile="${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg"
while getopts 'urnD' opt
do
case $opt in
@ -73,7 +54,6 @@ done
getConfig
sanityCheck
openDatabase
(( update )) && getFiles
@ -121,7 +101,7 @@ getdstfiles() {
;
SELECT "AtOM:NoMoreFiles";
'>&3
while read -u4 -r -d $'\0' line
while read -u4 line
do
if [[ $line == AtOM:NoMoreFiles ]]
then
@ -159,7 +139,7 @@ renameFile() {
fi
}
while read -u4 -r -d $'\0' line
while read -u4 line
do
if [[ $line == AtOM:NoMoreFiles ]]
then

View File

@ -1,18 +1,5 @@
#!/usr/bin/env bash
declare -r \
DOCDIR=%DOCDIR% \
LIBDIR=%LIBDIR% \
SHAREDIR=%SHAREDIR%
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
## Define exit codes
source "$SHAREDIR"/errorcodes
# config structures
declare -A \
destinationchannels \
@ -35,6 +22,17 @@ declare -A \
exit $EBASHVERS
}
declare -r \
DOCDIR=./doc \
LIBDIR=./lib \
SHAREDIR=./share
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
cffile="$HOME/.atom/atom.cfg"
LC_ALL=C
shopt -s extglob
@ -44,15 +42,6 @@ do
source "$function"
done
if ! [[ -f "${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg" ]] \
&& [[ -f "$HOME/.atom/atom.cfg" ]]
then
echo "Configuration found in legacy location $HOME/.atom/atom.cfg."\
"Migrating configuration and data to XDG standard"
xdgMigrate
fi
cffile="${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg"
while getopts 'D' opt
do
case $opt in
@ -61,7 +50,7 @@ do
done
getConfig
sanityCheck
openDatabase
sourcepath=''

View File

@ -1,18 +1,5 @@
#!/usr/bin/env bash
declare -r \
DOCDIR=%DOCDIR% \
LIBDIR=%LIBDIR% \
SHAREDIR=%SHAREDIR%
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
## Define exit codes
source "$SHAREDIR"/errorcodes
# config structures
declare -A \
destinationchannels \
@ -35,6 +22,17 @@ declare -A \
exit $EBASHVERS
}
declare -r \
DOCDIR=./doc \
LIBDIR=./lib \
SHAREDIR=./share
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
cffile="$HOME/.atom/atom.cfg"
LC_ALL=C
shopt -s extglob
@ -44,15 +42,6 @@ do
source "$function"
done
if ! [[ -f "${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg" ]] \
&& [[ -f "$HOME/.atom/atom.cfg" ]]
then
echo "Configuration found in legacy location $HOME/.atom/atom.cfg."\
"Migrating configuration and data to XDG standard"
xdgMigrate
fi
cffile="${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg"
while getopts 'Dr' opt
do
case $opt in
@ -62,45 +51,30 @@ do
done
getConfig
sanityCheck
openDatabase
echo -n "Checking for missing files... "
echo '
SELECT
destination_files.id,
destinations.name,
destination_files.filename
FROM destinations
INNER JOIN destination_files
ON destinations.id=destination_files.destination_id
WHERE filename IS NOT NULL;' >&3
echo 'SELECT id,filename FROM destination_files WHERE filename IS NOT NULL;' >&3
echo 'SELECT "AtOM:NoMoreFiles";' >&3
declare -a \
destination_names \
files
read -u4 -r -d $'\0' line
until [[ $line == AtOM:NoMoreFiles ]]
read -u4 filename
until [[ $filename == AtOM:NoMoreFiles ]]
do
id=${line%%::AtOM:SQL:Sep::*}
rest=${line#*::AtOM:SQL:Sep::}
destination_names[id]=${rest%%::AtOM:SQL:Sep::*}
rest=${rest#*::AtOM:SQL:Sep::}
files[id]=${rest}
read -u4 -r -d $'\0' line
files+=("$filename")
read -u4 filename
done
echo 'BEGIN TRANSACTION;' >&3
for index in "${!files[@]}"
echo -n "Checking for missing files... "
for filename in "${files[@]}"
do
destination=${destination_names[index]}
filename="${destinationpath[$destination]}/${files[index]}"
id=${filename%%::AtOM:SQL:Sep::*}
filename=${filename#*::AtOM:SQL:Sep::}
if ! [ -f "$filename" ]
then
echo -e "\r$filename\033[K"
((regen))&&Update destination_files last_change 0 <<<"id = $index"
((regen))&&Update destination_files last_change 0 <<<"id = $id"
echo -n "Checking for missing files... "
(( missing++ ))
fi
@ -109,6 +83,6 @@ done
echo 'COMMIT;' >&3
echo -e "\r${missing:-No} missing files\033[K"
echo -e "\r${missing:-0} missing files\033[K"
closeDatabase

View File

@ -1,18 +1,5 @@
#!/usr/bin/env bash
declare -r \
DOCDIR=%DOCDIR% \
LIBDIR=%LIBDIR% \
SHAREDIR=%SHAREDIR%
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
## Define exit codes
source "$SHAREDIR"/errorcodes
# config structures
declare -A \
destinationchannels \
@ -35,6 +22,17 @@ declare -A \
exit $EBASHVERS
}
declare -r \
DOCDIR=./doc \
LIBDIR=./lib \
SHAREDIR=./share
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
cffile="$HOME/.atom/atom.cfg"
LC_ALL=C
shopt -s extglob
@ -44,15 +42,6 @@ do
source "$function"
done
if ! [[ -f "${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg" ]] \
&& [[ -f "$HOME/.atom/atom.cfg" ]]
then
echo "Configuration found in legacy location $HOME/.atom/atom.cfg."\
"Migrating configuration and data to XDG standard"
xdgMigrate
fi
cffile="${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg"
while getopts 'rD' opt
do
case $opt in
@ -62,25 +51,28 @@ do
done
getConfig
sanityCheck
openDatabase
checkwanted() {
Select id <<<"filename = $1"
}
for destination in "${!destinationpath[@]}"
do
echo -ne "\rScanning destination $destination... \033[K"
while read -r -d $'\0' filename
while read -r filename
do
sqlfile=${filename//$'\n'/::AtOM:NewLine:SQL:Inline::}
if ! Select destination_files id \
>/dev/null \
<<<"filename = ${sqlfile#${destinationpath["$destination"]}/}"
<<<"filename = $filename"
then
echo -e $'\r'"$filename\033[K"
(( remove )) && rm "$filename"
(( remove )) && rm -f "$filename"
echo -n "Scanning destination $destination... "
fi
progressSpin
done < <(find "${destinationpath["$destination"]}" -type f -print0)
done < <(find "${destinationpath["$destination"]}" -type f)
done
echo -en "\r\033[K"

View File

@ -1,17 +1,6 @@
#!/usr/bin/env bash
declare -r \
DOCDIR=%DOCDIR% \
LIBDIR=%LIBDIR% \
SHAREDIR=%SHAREDIR%
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
## Define exit codes
source "$SHAREDIR"/errorcodes
#!/bin/bash
# config structures
declare -A \
@ -35,26 +24,29 @@ declare -A \
exit $EBASHVERS
}
declare -r \
DOCDIR=./doc \
LIBDIR=./lib \
SHAREDIR=./share
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
cffile="$HOME/.atom/atom.cfg"
LC_ALL=C
shopt -s extglob
source "$SHAREDIR"/id3genres
source ./share/id3genres
for function in "$LIBDIR"/*/*
do
source "$function"
done
if ! [[ -f "${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg" ]] \
&& [[ -f "$HOME/.atom/atom.cfg" ]]
then
echo "Configuration found in legacy location $HOME/.atom/atom.cfg."\
"Migrating configuration and data to XDG standard"
xdgMigrate
fi
cffile="${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg"
args="$@"
while [ -n "$1" ]
do
@ -74,7 +66,6 @@ do
'-M') show+=(types) ;;
'-N') show+=(tracktotals) ;;
'-p') show+=(performers) ;;
'-r') show+=(releasecountries) ;;
'-s') show+=(rates) ;;
'-S') show+=(size)
length[count]=5 ;;
@ -123,7 +114,6 @@ done
-d Disc
-g Genre
-p Performer
-r Release Country
-N Track total
-t Title
-y Year
@ -140,7 +130,7 @@ done
}
getConfig
sanityCheck
openDatabase
columns="${show[@]//*/-}"
@ -274,7 +264,7 @@ fi
echo 'SELECT IFNULL(
(SELECT last_seen FROM source_files ORDER BY last_seen DESC LIMIT 1),
0);' >&3
read -u4 -r -d $'\0' lastupdate
read -u4 lastupdate
if ! [[ "$output" == - ]]
then
@ -283,17 +273,17 @@ fi
cat <<-EOBrag
# Generated by AtOM's createindex toy.
# https://framagit.org/atom/AtOM/
# (C) 2012-2025 Vincent Riquer (GPL-3)
# https://gitorious.org/atom
# (C) 2012-2013 Vincent Riquer (GPL-3)
#
# $0 $args
#
# Last database update: $(printf "%(%x %X)T" "$lastupdate")
# Last database update: $(date -d @$lastupdate +'%x %X')
EOBrag
printDate() {
printf "%("${timeformat:-%x %X}")T" "$1"
date -d"@$1" +"${timeformat:-%x %X}"
}
for index in ${!show[@]}
@ -315,7 +305,6 @@ do
oldtimestamp) info='Last modified' ;;
types) info='Format' ;;
performers) info='Performer' ;;
releasecountries) info='Country' ;;
rates) info='Sample rate' ;;
titles) info='Title' ;;
tracktotals) info='Track total' ;;
@ -349,7 +338,6 @@ SELECT
tags.disc,
tags.genre,
tags.performer,
tags.releasecountry,
tags.title,
tags.track,
tags.year,
@ -369,11 +357,11 @@ COLLATE NOCASE;
SELECT "AtOM:NoMoreFiles";' >&3
read -u4 -r -d $'\0' line
read -u4 line
until [[ $line == AtOM:NoMoreFiles ]]
do
files+=("$line")
read -u4 -r -d $'\0' line
read -u4 line
done
for line in "${files[@]}"
@ -408,8 +396,6 @@ do
rest="${rest#*::AtOM:SQL:Sep::}"
performer="${rest%%::AtOM:SQL:Sep::*}"
rest="${rest#*::AtOM:SQL:Sep::}"
releasecountry="${rest%%::AtOM:SQL:Sep::*}"
rest="${rest#*::AtOM:SQL:Sep::}"
title="${rest%%::AtOM:SQL:Sep::*}"
rest="${rest#*::AtOM:SQL:Sep::}"
track="${rest%%::AtOM:SQL:Sep::*}"
@ -453,10 +439,7 @@ do
expr2='(,|$)'
if ! [[ $channelss =~ $expr1"$channels"$expr2 ]]
then
if [[ -n "$channels" ]]
then
:
elif [ -n "$channelss" ] \
if [ -n "$channelss" ] \
&& (( channels < ${channelss%%,*} ))
then
channelss="$channels,$channelss"
@ -466,10 +449,7 @@ do
fi
if ! [[ $rates =~ $expr1"$rate"$expr2 ]]
then
if [[ -n "$rate" ]]
then
:
elif [ -n "$rates" ] \
if [ -n "$rates" ] \
&& (( rate < ${rates%%,*} ))
then
rates="$rate,$rates"
@ -479,10 +459,7 @@ do
fi
if [ -n "$depth" ] && ! [[ $depths =~ $expr1"$depth"$expr2 ]]
then
if [[ -n "$depth" ]]
then
:
elif [ -n "$depths" ] \
if [ -n "$depths" ] \
&& (( depth < ${depths%%,*} ))
then
depths="$depth,$depths"
@ -546,13 +523,6 @@ do
[ -n "$performer" ] \
&& performers+="${performers+,}$performer"
fi
if ! [[ $releasecountries =~ $expr1"$releasecountry"$expr2 ]]
then
[ -z "$releasecountries" ] \
&& unset releasecountries
[ -n "$releasecountry" ] \
&& releasecountries+="${releasecountries+,}$releasecountry"
fi
if ! [[ $titles =~ $expr1"$title"$expr2 ]]
then
[ -z "$titles" ] \
@ -579,10 +549,10 @@ do
then
printline
fi
unset bitrates depths rates
unset bitrates
channelss="$channels"
(( rate )) && rates="$rate"
(( depth )) && depths="$depth"
rates="$rate"
depths="$depth"
types="$type"
albumartists="$albumartist"
albums="$album"
@ -591,7 +561,6 @@ do
discs="$disc"
genres="$genre"
performers="$performer"
releasecountries="$releasecountry"
titles="$title"
tracktotals="$tracktotal"
years="$year"
@ -675,7 +644,7 @@ echo '
SELECT "AtOM:NoMoreFiles";' >&3
read -u4 -r -d $'\0' line
read -u4 line
until [[ $line == AtOM:NoMoreFiles ]]
do
artist="${line%%::AtOM:SQL:Sep::*}"
@ -685,7 +654,7 @@ do
artists+=( "$artist" )
maxcountlen=$(( ${#count} > maxcountlen ? ${#count} : maxcountlen ))
maxartistlen=$(( ${#artist} > maxartistlen ? ${#artist} : maxartistlen ))
read -u4 -r -d $'\0' line
read -u4 line
done
head=$(
printf "| # | %'${maxcoutlen}s | %-${maxartistlen}s |" \
@ -714,7 +683,7 @@ echo '
FROM source_files
INNER JOIN mime_types
ON source_files.mime_type=mime_types.id;' >&3
read -u4 -r -d $'\0' line
read -u4 line
totalcount="${line%%::AtOM:SQL:Sep::*}"
maxcountlen=$(printf "%'i" $totalcount)
maxcountlen=${#maxcountlen}
@ -743,7 +712,7 @@ do
INNER JOIN mime_types
ON source_files.mime_type=mime_types.id
WHERE mime_text LIKE "'"$format"'";' >&3
read -u4 -r -d $'\0' line
read -u4 line
count="${line%%::AtOM:SQL:Sep::*}"
rest="${line#*::AtOM:SQL:Sep::}::AtOM:SQL:Sep::"
size="${rest%%::AtOM:SQL:Sep::*}"

View File

@ -1,18 +1,5 @@
#!/usr/bin/env bash
declare -r \
DOCDIR=%DOCDIR% \
LIBDIR=%LIBDIR% \
SHAREDIR=%SHAREDIR%
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
## Define exit codes
source "$SHAREDIR"/errorcodes
# config structures
declare -A \
destinationchannels \
@ -35,6 +22,17 @@ declare -A \
exit $EBASHVERS
}
declare -r \
DOCDIR=./doc \
LIBDIR=./lib \
SHAREDIR=./share
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
cffile="$HOME/.atom/atom.cfg"
LC_ALL=C
shopt -s extglob
@ -44,15 +42,6 @@ do
source "$function"
done
if ! [[ -f "${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg" ]] \
&& [[ -f "$HOME/.atom/atom.cfg" ]]
then
echo "Configuration found in legacy location $HOME/.atom/atom.cfg."\
"Migrating configuration and data to XDG standard"
xdgMigrate
fi
cffile="${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg"
while getopts 'fm:o:p:*:uD' opt
do
case $opt in
@ -78,7 +67,7 @@ do
done
getConfig
sanityCheck
openDatabase
if (( update ))
@ -131,11 +120,11 @@ echo ') ORDER BY bitrate;' >&3
echo 'SELECT "AtOM:NoMoreFiles";' >&3
read -u4 -r -d $'\0' line
read -u4 line
until [[ $line == AtOM:NoMoreFiles ]]
do
echo "${line//::AtOM:SQL:Sep::/$'\t'}"
read -u4 -r -d $'\0' line
read -u4 line
done
closeDatabase

View File

@ -1,18 +1,5 @@
#!/usr/bin/env bash
declare -r \
DOCDIR=%DOCDIR% \
LIBDIR=%LIBDIR% \
SHAREDIR=%SHAREDIR%
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
## Define exit codes
source "$SHAREDIR"/errorcodes
# config structures
declare -A \
destinationchannels \
@ -36,6 +23,17 @@ declare -A \
exit $EBASHVERS
}
declare -r \
DOCDIR=./doc \
LIBDIR=./lib \
SHAREDIR=./share
declare -r \
exampleconf=$DOCDIR/example.cfg \
schema=$SHAREDIR/schema.sql \
\
oldIFS="$IFS"
cffile="$HOME/.atom/atom.cfg"
LC_ALL=C
shopt -s extglob
@ -45,15 +43,6 @@ do
source "$function"
done
if ! [[ -f "${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg" ]] \
&& [[ -f "$HOME/.atom/atom.cfg" ]]
then
echo "Configuration found in legacy location $HOME/.atom/atom.cfg."\
"Migrating configuration and data to XDG standard"
xdgMigrate
fi
cffile="${XDG_CONFIG_HOME:-$HOME/.config}/AtOM/atom.cfg"
while getopts 'AlacdgptnNyuD' opt
do
case $opt in
@ -97,7 +86,7 @@ done
}
getConfig
sanityCheck
openDatabase
if (( update ))
@ -149,11 +138,11 @@ cat >&3 <<-EOSelect
SELECT "AtOM:NoMoreFiles";
EOSelect
read -u4 -r -d $'\0' line
read -u4 line
until [[ $line == AtOM:NoMoreFiles ]]
do
lines+=( "$line" )
read -u4 -r -d $'\0' line
read -u4 line
done
for line in "${lines[@]}"