<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Thank you for the very useful comments. However,</div><div class="gmail_default" style="font-family:monospace,monospace">I disagree with you about the RE language. While</div><div class="gmail_default" style="font-family:monospace,monospace">I agree all RE experts don't need that, when I was</div><div class="gmail_default" style="font-family:monospace,monospace">hiring and gave some software to a new hire (whether</div><div class="gmail_default" style="font-family:monospace,monospace">an experienced programmer or a recent college grad)</div><div class="gmail_default" style="font-family:monospace,monospace">simply handing over huge RE's to my new hire was</div><div class="gmail_default" style="font-family:monospace,monospace">a daunting task to that person. I wrote that stuff</div><div class="gmail_default" style="font-family:monospace,monospace">that way to help remind me and anyone who might</div><div class="gmail_default" style="font-family:monospace,monospace">use the python program.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">I don't claim success. It does help me.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">When you say '{1}' is redundant, I think I did that</div><div class="gmail_default" style="font-family:monospace,monospace">to avoid any possibility of conflicts with the</div><div class="gmail_default" style="font-family:monospace,monospace">next string that is</div><div class="gmail_default" style="font-family:monospace,monospace">concatentated to the <b>Y_</b> (e.g. '*' or '+' or '{4,7}').</div><div class="gmail_default" style="font-family:monospace,monospace">I am embarrassed I did not communicate that</div><div class="gmail_default" style="font-family:monospace,monospace">in the code. I had to think about it for a couple of hours</div><div class="gmail_default" style="font-family:monospace,monospace">before I recalled the "why". I will fix that.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"> (it would be difficult to discuss</div><div class="gmail_default" style="font-family:monospace,monospace"> this RE if I had to write</div><div class="gmail_default" style=""><font face="monospace, monospace"> </font><span style="font-family:Arial,Helvetica,sans-serif">"(19\d\d|20[01]\d|202" + "[0-" + lastYearRE + "]" + ")</span></div><div class="gmail_default" style=""><font face="monospace"> rather than just </font><b style=""><font face="arial, sans-serif">Y_</font></b><font face="monospace">).</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">My initial thoughts</font></div><div class="gmail_default" style=""><font face="monospace">on naming were </font><span style="font-family:monospace">I wanted the definition to be defined</span></div><div class="gmail_default" style=""><span style="font-family:monospace">in exactly one place </span><span style="font-family:monospace">in the software.</span></div><div class="gmail_default" style=""><span style="font-family:monospace">Python and the BTL folks told me to never</span></div><div class="gmail_default" style=""><span style="font-family:monospace">use a constant in code. Always name it.</span></div><div class="gmail_default" style=""><span style="font-family:monospace">Hence, I gave it a name. Each name might</span></div><div class="gmail_default" style=""><span style="font-family:monospace">be used in multiple places. They might be imported.</span></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">You are correct, the expression is unbalanced. I tried</font></div><div class="gmail_default" style=""><font face="monospace">to remove the text2bytes(lastYearRE<b><font size="4">)</font></b> call so the expression in</font></div><div class="gmail_default" style=""><font face="monospace">this email was all text. I failed to remove the trailing <b><font size="4">)</font></b> when</font></div><div class="gmail_default" style=""><font face="monospace">I removed the call to text2bytes(). My hasty transcriptions</font></div><div class="gmail_default" style=""><font face="monospace">might have produced similar errors in my email.</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">Recall, my focus was </font><span style="font-family:monospace">on any file of any size.</span></div><div class="gmail_default" style=""><span style="font-family:monospace">I'm on Windows 10 and an m1 MacBook.</span></div><div class="gmail_default" style=""><font face="monospace">Python works on both. I don't have</font></div><div class="gmail_default" style=""><font face="monospace">a Linux machine or enough desktop space to</font></div><div class="gmail_default" style=""><font face="monospace">host one. I'm also mildly fed-up with</font></div><div class="gmail_default" style=""><font face="monospace">virtual machines. </font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">Friedl taught me one thing. Most</font></div><div class="gmail_default" style=""><font face="monospace">RE implementations are different. I'm trying</font></div><div class="gmail_default" style=""><font face="monospace">to write a program that I could give</font></div><div class="gmail_default" style=""><font face="monospace">to anyone and could reliably find a date (an RE) in</font></div><div class="gmail_default" style=""><font face="monospace">any file. YYYY, MM, DD, HR, MI, SE, TH are words</font></div><div class="gmail_default" style=""><font face="monospace">my user could use in the command line or in</font></div><div class="gmail_default" style=""><font face="monospace">an options dialog. LAT and LON might also be</font></div><div class="gmail_default" style=""><font face="monospace">possibilities. CST, EST, MST, PST, ... also.</font></div><div class="gmail_default" style=""><font face="monospace">A 500 gigabyte archive or directory/folder</font></div><div class="gmail_default" style=""><font face="monospace">of pictures and movies would be</font></div><div class="gmail_default" style=""><font face="monospace">a great test target.</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">I very much appreciate your comments. If this</font></div><div class="gmail_default" style=""><font face="monospace">discussion is boring to others, I would be happy</font></div><div class="gmail_default" style=""><font face="monospace">to take it to emails.</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">I like your program. My experience</font></div><div class="gmail_default" style=""><font face="monospace">with RE, grep, python, and sed suggests that</font></div><div class="gmail_default" style=""><font face="monospace">anything but gnu grep and sed might not work due to the</font></div><div class="gmail_default" style=""><font face="monospace">different implementations.</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">I've been out of the </font><span style="font-family:monospace">Unix software business</span></div><div class="gmail_default" style=""><span style="font-family:monospace">for 30 years after starting work at BTL in the 1970s</span></div><div class="gmail_default" style=""><span style="font-family:monospace">and working on Version 6. I didn't know "printf" was now</span></div><div class="gmail_default" style=""><span style="font-family:monospace">built into bash! That was a surprise. It's an incremental</span></div><div class="gmail_default" style=""><span style="font-family:monospace">improvement, but doesn't compare with f-strings in python.</span></div><div class="gmail_default" style=""><span style="font-family:monospace"><i><font size="1">The interactive interpreter for python should have</font></i></span></div><div class="gmail_default" style=""><span style="font-family:monospace"><i><font size="1">a "bash" mode?!</font></i></span></div><div class="gmail_default" style=""><span style="font-family:monospace"><br></span></div><div class="gmail_default" style=""><font face="monospace">Does grep use a memory mapped file for its search, thereby</font></div><div class="gmail_default" style=""><font face="monospace">avoiding all buffering boundaries? That too, would</font></div><div class="gmail_default" style=""><font face="monospace">be new information to me. The additional complexity</font></div><div class="gmail_default" style=""><font face="monospace">of dealing with buffering is more than annoying.</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">Do you have any thoughts on how to verify</font></div><div class="gmail_default" style=""><font face="monospace">a program that uses RE's. I've given no thought</font></div><div class="gmail_default" style=""><font face="monospace">until now. My first thought for dates would be</font></div><div class="gmail_default" style=""><font face="monospace">to write a separate module that simply searched</font></div><div class="gmail_default" style=""><font face="monospace">through the file looking for 4 numbers in a row</font></div><div class="gmail_default" style=""><font face="monospace">without using RE's, recording the offsets and 16 characters</font></div><div class="gmail_default" style=""><font face="monospace">after and 1 character before in a python list of (offset,str)</font></div><div class="gmail_default" style=""><font face="monospace">of tuples, </font><span style="font-family:arial,sans-serif;font-style:italic">ddddList</span><span style="font-family:arial,sans-serif;font-style:italic">, </span><font face="monospace">and using </font><font face="arial, sans-serif"><i>dddd</i></font><i style="font-family:arial,sans-serif">List</i></div><div class="gmail_default" style=""><span style="font-family:monospace">as a proxy for the entire file. I could then</span></div><div class="gmail_default" style=""><font face="monospace">aim my RE's at </font><font face="arial, sans-serif"><i><b>ddddList</b></i></font><font face="monospace">. </font><b style=""><font face="comic sans ms, sans-serif">[A list of tuples in python</font></b></div><div class="gmail_default" style=""><b style=""><font face="comic sans ms, sans-serif">is wonderful! !]</font></b><font face="monospace"> It seems to me '*' and '+' and {x,y} are the performance</font></div><div class="gmail_default" style=""><font face="monospace">hogs in RE's. My RE's avoid them. One pass, I think, should</font></div><div class="gmail_default" style=""><font face="monospace">suffice. What do you think? I haven't "archived" my 350 GB</font></div><div class="gmail_default" style=""><font face="monospace">of pictures and movies, but one pass over all files therein</font></div><div class="gmail_default" style=""><font face="monospace">ought to suffice, right? Two different programs that use different</font></div><div class="gmail_default" style=""><font face="monospace">algorithms should be pretty good proof of correctness wouldn't</font></div><div class="gmail_default" style=""><font face="monospace">you think?</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">My RE's have no stars or pluses. If there is a mismatch before</font></div><div class="gmail_default" style=""><font face="monospace">a match, give up and move on.</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">On my Windows 10 machine, I have cygwin.</font></div><div class="gmail_default" style=""><font face="monospace">Microsoft says my CPU doesn't have a TPM and</font></div><div class="gmail_default" style=""><font face="monospace">the specific Intel Core I7 on my system is not</font></div><div class="gmail_default" style=""><font face="monospace">supported so Windows 11 is not happening.</font></div><div class="gmail_default" style=""><font face="monospace">Microsoft is DOS personified.</font></div><div class="gmail_default" style=""><font face="monospace"> (An unkind editorial remark about the low</font></div><div class="gmail_default" style=""><font face="monospace"> quality of software coming from Microsoft.)</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace">Anyway, I thank you again for your patience with me</font></div><div class="gmail_default"><font face="monospace">and your observations. I value your views and the</font></div><div class="gmail_default"><font face="monospace">other views I've seen here on <a href="mailto:coff@tuhs.org">coff@tuhs.org</a>.</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace">I welcome all input to my education and will share</font></div><div class="gmail_default"><font face="monospace">all I have done so far with anyone who wants to</font></div><div class="gmail_default"><font face="monospace">collaborate, test, or is just curious.</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace"> GOAL: run python program from an at-cost thumb drive that:</font></div><div class="gmail_default"><font face="monospace"> reaps all media files from a user specified</font></div><div class="gmail_default"><font face="monospace"> directory/folder tree and</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace"> Adds files to the thumb drive.</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><div class="gmail_default"><font face="monospace"> <b>Adds files</b> means</font></div><div class="gmail_default"><font face="monospace"> Original file system is untouched</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace"> Adds only unique files (hash codes are unique)</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace"> Creates on the thumb drive a relative directory</font></div><div class="gmail_default"><font face="monospace"> wherein the </font><span style="font-family:monospace">original file was found</span></div><div class="gmail_default"><span style="font-family:monospace"><br></span></div><div class="gmail_default"><font face="monospace"> Prepends a "YYYY-MM-DD-" string to the filename</font></div><div class="gmail_default"><font face="monospace"> if one can be found (EXIF is great shortcut).</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace"> Copies</font></div><div class="gmail_default"><font face="monospace"> srcroot/relative_path/oldfilename</font></div><div class="gmail_default"><font face="monospace"> to</font></div><div class="gmail_default"><font face="monospace"> thumbdrive/relative_path/YYYY-MM-DD-oldfilename</font></div><div class="gmail_default"><font face="monospace"> or</font></div><div class="gmail_default"><font face="monospace"> thumbdrive/relative_path/0000-oldfilename.</font></div></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace"> Can also incrementally add new files by just</font></div><div class="gmail_default"><font face="monospace"> scanning anywhere in any other computer</font></div><div class="gmail_default"><font face="monospace"> file system or any other computer.</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace"> Must work on Mac, Windows, and Linux</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace">What I have is a working prototype. It works</font></div><div class="gmail_default"><font face="monospace">on Mac and Windows. It doesn't do the</font></div><div class="gmail_default"><font face="monospace">date thing very well, and there are other shortcomings.</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace">I have delivered exactly one Christmas present to my favorite person</font></div><div class="gmail_default"><font face="monospace">in the world - a 400 GB SSD drive with all our pictures and media</font></div><div class="gmail_default"><font face="monospace">we have ever taken. The next things are </font><span style="font-family:monospace">to </span><b style="font-family:monospace">add </b><span style="font-family:monospace">more media</span></div><div class="gmail_default"><span style="font-family:monospace">and</span><span style="font-family:monospace"> <b>re-unique-ify</b> </span><span style="font-family:monospace">(check) what is already present on the SSD drive</span></div><div class="gmail_default"><span style="font-family:monospace">and <b>improve the proper choice of "YYYY-MM-DD-" prefix</b> to</span></div><div class="gmail_default"><span style="font-family:monospace">filenames.</span></div><div class="gmail_default" style=""><br></div><div class="gmail_default" style=""><font face="monospace">I am retired and this is fun.</font></div><div class="gmail_default" style=""><font face="monospace">I'm too old to want to get rich.</font></div><div class="gmail_default" style=""><font face="monospace"><br></font></div><div class="gmail_default" style=""><font face="monospace">Ed Bradford</font></div><div class="gmail_default" style=""><font face="monospace">Pflugerville, TX</font></div><div class="gmail_default" style=""><font face="monospace"><a href="mailto:egbegb2@gmail.com">egbegb2@gmail.com</a></font></div><div class="gmail_default" style=""><br></div><div class="gmail_default" style=""><br></div><div class="gmail_default" style=""><br></div><div class="gmail_default" style=""><font face="monospace"><br></font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 7, 2023 at 5:40 AM Ralph Corderoy <<a href="mailto:ralph@inputplus.co.uk">ralph@inputplus.co.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Ed,<br>
<br>
> I have made an attempt to make my RE stuff readable and supportable.<br>
<br>
Readable to you, which is fine because you're the prime future reader.<br>
But it's less readable than the regexp to those that know and read them<br>
because of the indirection introduced by the variables. You've created<br>
your own little language of CAPITALS rather than the lingua franca of<br>
regexps. :-)<br>
<br>
> Machine language was unreadable and then along came assembly language.<br>
> Assembly language was unreadable, then came higher level languages.<br>
<br>
Each time the original language was readable because practitioners had<br>
to read and write it. When its replacement came along, the old skill<br>
was no longer learnt and the language became ‘unreadable’.<br>
<br>
> So far, I can do that for this RE program that works for small files,<br>
> large files, binary files and text files for exactly one pattern:<br>
> YYYY[-MM-DD]<br>
> I constructed this RE with code like this:<br>
> # ymdt is YYYY-MM-DD RE in text.<br>
> # looking only for 1900s and 2000s years and no later than today.<br>
> _YYYY = "(19\d\d|20[01]\d|202" + "[0-" + lastYearRE) + "]" + "){1}"<br>
<br>
‘{1}’ is redundant.<br>
<br>
> # months<br>
> _MM = "(0[1-9]|1[012])"<br>
> # days<br>
> _DD = "(0[1-9]|[12]\d|3[01])"<br>
> ymdt = _YYYY + '[' + _INTERNALSEP +<br>
> _MM +<br>
> _INTERNALSEP +<br>
> ']'{0,1)<br>
<br>
I think we're missing something as the ‘'['’ is starting a character<br>
class which is odd for wrapping the month and the ‘{0,1)’ doesn't have<br>
matching brackets and is outside the string.<br>
<br>
BTW, ‘{0,1}’ is more readable to those who know regexps as ‘?’.<br>
<br>
> For the whole file, RE I used<br>
> ymdthf = _FRSEP + ymdt + _BASEP<br>
> where FRSEP is front separator which includes<br>
> a bunch of possible separators, excluding numbers and letters, or-ed<br>
> with the up arrow "beginning of line" RE mark.<br>
<br>
It sounds like you're wanting a word boundary; something provided by<br>
regexps. In Python, it's ‘\b’.<br>
<br>
>>> re.search(r'\bfoo\b', 'endfoo foostart foo ends'),<br>
(<re.Match object; span=(16, 19), match='foo'>,)<br>
<br>
Are you aware of the /x modifier to a regexp which ignores internal<br>
whitespace, including linefeeds? This allows a large regexp to be split<br>
over lines. There's a comment syntax too. See<br>
<a href="https://docs.python.org/3/library/re.html#re.X" rel="noreferrer" target="_blank">https://docs.python.org/3/library/re.html#re.X</a><br>
<br>
GNU grep isn't too shabby at looking through binary files. I can't use<br>
/x with grep so in a bash script, I'd do it manually. \< and \> match<br>
the start and end of a word, a bit like Python's \b.<br>
<br>
re='<br>
.?\<<br>
(19[0-9][0-9]|20[01][0-9]|202[0-3])<br>
(<br>
([-:._])<br>
(0[1-9]|1[0-2])<br>
\3<br>
(0[1-9]|[12][0-9]|3[01])<br>
)?<br>
\>.?<br>
'<br>
re=${re//$'\n'/}<br>
re=${re// /}<br>
<br>
printf '%s\n' 2001-04-01,1999_12_31 1944.03.01,1914! 2000-01.01 >big-binary-file<br>
LC_ALL=C grep -Eboa "$re" big-binary-file | sed -n l<br>
<br>
which gives<br>
<br>
0:2001-04-01,$<br>
11:1999_12_31$<br>
22:1944.03.01,$<br>
33:1914!$<br>
39:2000-$<br>
<br>
showing:<br>
<br>
- the byte offset within the file of each match,<br>
- along with the any before and after byte if it's not a \n and not<br>
already matched, just to show the word-boundary at work,<br>
- with any non-printables escaped into octal by sed.<br>
<br>
> I thought I was on the COFF mailing list.<br>
<br>
I'm sending this to just the list.<br>
<br>
> I received this email by direct mail to from Larry.<br>
<br>
Perhaps your account on the list is configured to not send you an email<br>
if it sees your address in the header's fields.<br>
<br>
-- <br>
Cheers, Ralph.<br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><font face="'courier new', monospace"><span style="font-weight:900"><div>Advice is judged by results, not by intentions.</div><div> Cicero</div></span></font><div><br></div></div>