[clc-devel] RFC: a script to check availability of updates
Hi everybody, I'm putting together a set of scripts to automatically check for new releases of packages. I currently maintain few ports in CLC, more are planned and I needed a way to automate the search for new releases. I attach an archive with a preliminary system of main script + tests + config file. Please refer to the README file for detailed description and other infos. Comments and contributions are welcome, inspite I won't have many time in the next 2 weeks (exams approaching). Regards, Simone Rota
On Sun, 7 Sep 2003 23:16:43 +0200 Simone Rota <sip@varlock.com> wrote: (I hate auto-replying...)
Comments and contributions are welcome, inspite I won't have many time in the next 2 weeks (exams approaching).
I meant: even if I won't have many time in the next 2 weeks (exams approaching).
On Sun, Sep 07, 2003 at 11:16:43PM +0200, Simone Rota wrote:
Hi everybody,
Hi Simone, hi all,
I'm putting together a set of scripts to automatically check for new releases of packages.
hmm, just for your information, maybe it's not known by everyone. You can "subscribe to new releases" for an unlimited number of projects on freshmeat. Create a account there, subscribe your progs and you will get an email for every new release. This works for nearly all of my ports. Greetings Jürgen -- juergen.daubert@t-online.de
On Sun, Sep 07, 2003 at 11:16:43PM +0200, Simone Rota wrote:
Hi everybody,
Hi Simone, hi all,
I'm putting together a set of scripts to automatically check for new releases of packages.
hmm, just for your information, maybe it's not known by everyone. You can "subscribe to new releases" for an unlimited number of projects on freshmeat. Create a account there, subscribe your progs and you will get an email for every new release. This works for nearly all of my ports. I agree the freshmeat notification is a pretty good thing, OTOH some
Hi Jürgen, On Mon, Sep 08, 2003 at 07:41:25 +0200, juergen.daubert@t-online.de wrote: ports need special attention (e.g. the j2sdk binary takes a while until it's found on a public ftp site), and therefore I like to have one utility able to check both freshmeat and "other" sites :-) (for now I use a homebrewn script, but I'll have a look at Simone's). Anyway, to each his own ;-) Kind regards, Johannes -- Johannes Winkelmann mailto:jw@tks6.net Biel, Switzerland http://jw.tks6.net
On Mon, Sep 08, 2003 at 10:48:37AM +0200, Johannes Winkelmann wrote: [...]
I agree the freshmeat notification is a pretty good thing, OTOH some ports need special attention (e.g. the j2sdk binary takes a while until it's found on a public ftp site), and therefore I like to have one utility able to check both freshmeat and "other" sites :-) (for now I use a homebrewn script, but I'll have a look at Simone's).
I've been using some stupid script that looks at .listing files and doesn't work very well (fortunately there's the gnome ftp-release-list). Anyway, if Simone's scripts work well perhaps the CLC could implement a system of notifying maintainers/packagers when they need to update their ports? That would be super cool. :-) //rrm3 -- The end of the human race will be that it will eventually die of civilization. -- Ralph Waldo Emerson
On Mon, Sep 08, 2003 at 10:48:37AM +0200, Johannes Winkelmann wrote: [...]
I agree the freshmeat notification is a pretty good thing, OTOH some ports need special attention (e.g. the j2sdk binary takes a while until it's found on a public ftp site), and therefore I like to have one utility able to check both freshmeat and "other" sites :-)
Yes, of course ! My intention was not to query Simone's script but to point on the freshmeat possibilities ;) I'll test it today for the progs which does not announce new releases on freshmeat. kind regards Jürgen -- juergen.daubert@t-online.de
On Mon, 8 Sep 2003 07:41:25 +0200 juergen.daubert@t-online.de wrote:
Hi Simone, hi all,
Hi Juergen,
You can "subscribe to new releases" for an unlimited number of projects on freshmeat. Create a account there, subscribe your progs and you will get an email for every new release. This works for nearly all of my ports.
Thank you for pointing out this hint. I already have an account at freshmeat, but never thought of subscribing to new releases. Sometimes the solution of a problem is so simple but I can't see it! Since I like to waste my time, I think I'll go on with the script, to handle ports not listed on freshmeat, or just to have some fun :)
Greetings Jürgen
Regards, Simone
On Sun, Sep 07, 2003 at 11:16:43PM +0200, Simone Rota wrote:
Hi everybody,
Hi Simone, hi all,
I'm putting together a set of scripts to automatically check for new releases of packages.
I've checked your script and and written my own, because I'm not satisfied with the overall performance when checking a large number of web pages. The attached ruby script has, hopefully, some improvements: - it's multi-threading, a configurable number of threads fetches the pages parallel to each other. Default is a max. of 20 threads, this can be changed at the top of the script (variable threads_max). - a regular expression can be applied to each page, only the matching part is used for comparison and saved for later use. For example: 'apache md5 http://www.apache.de/dist/httpd/ httpd-2[0-9.]*' This is usefully if e.g. daily snapshots are one the same page. - a kind of macro-expansion can be used in the configuration file, e.g you can define '@sf@ http://unc.dl.sourceforge.net' and use later a configuration line like: 'pure-ftpd md5 @sf@/pureftpd/' - restricting the pages to process is possible by applying a regexp on the command-line, e.g. 'ck4up ruby' will only check configuration lines which contains the string ruby. - data's are stored in a gdbm database '~/ck4up/ck4up.dbm', this is, of course, not a improvement but a feature ;) The included sample configuration file is somewhat commented and _MUST_ be manually installed as '~/ck4up/ck4up.conf'. No other docu actually. A 'ck4up -h' gives some help about command-line switches. I've included also a little bash-script (ck4up.sh), which starts mozilla with the different pages, or open them in new tabs, if mozilla is already running. I'm curious about your feedback, and, hopefully, this script will be useful for someone other. I know, ruby isn't a very popular scripting language right now, but give it a try, it's a wonderful and very lightweight language (~5M on my hd). And, maybe, you've it installed already, because you're using yapo ;) kind regards Jürgen -- juergen.daubert@t-online.de
On Sat, 13 Sep 2003 18:56:16 +0200 juergen.daubert@t-online.de wrote:
I've checked your script and and written my own, because I'm not satisfied with the overall performance when checking a large number of web pages. The attached ruby script has, hopefully, some improvements:
[cut] I wrote checkup as a sort of proof-of-concept, mainly to propose the mutliple tests idea, giving the user the ability to choose the most appropriate test for a certain port. I know the actual code sucks ;) It's nice to see that at least the idea of a checkup script is interesting and someone is working on such a tool. I'll definitly try your script.
I know, ruby isn't a very popular scripting language right now, but give it a try, it's a wonderful and very lightweight language (~5M on my hd). And, maybe, you've it installed already, because you're using yapo ;)
My original idea was to use python for the script, let the flame begin! (just joking)
kind regards Jürgen
Regards, Simone
On Sat, Sep 13, 2003 at 07:29:12PM +0200, Simone Rota wrote:
My original idea was to use python for the script, let the flame begin! (just joking)
hehe, some times agout I was a python fan also, but I've read somewhere "ruby is what python should be", and I must agree ;) Greetings Jürgen -- juergen.daubert@t-online.de
Hi again, while looking through the source-location of the gnome ports, I've seen that it could be handy to have a system macro '@NAME@' to simplify the generation and maintenance of ck4up.conf. You can now use the following definitions: '@GNOME@ http://ftp.gnome.org/pub/gnome/sources/@NAME@/2.4/' 'gnome-panel md5 @GNOME@' hopefully not bugging you and kind regards Jürgen -- juergen.daubert@t-online.de
Hi all, just in case someone is using it, attached a new version of the ck4up utility. No fancy new features, but code-cleanup and the possibility to nest "macro" definitions. I've included a macro @FM@ in the sample configuration file, which makes use of the freshmeat project XML database. With this, it's very lazy to add a line like 'crux md5 @FM@' to ck4up.conf and everything is done. Thanks to Johannes for pointing me on this nice freshmeat feature. kind regards Jürgen -- juergen.daubert@t-online.de
Hi Jürgen, On Fri, Oct 10, 2003 at 20:09:01 +0200, Juergen Daubert wrote:
Hi all,
just in case someone is using it, attached a new version of the ck4up utility. No fancy new features, but code-cleanup and the possibility to nest "macro" definitions. Great. I'm using ck4up to check for updates and it works perfectly. I'd even suggest to create a port for this as monitoring a web site for changes is a task others probably want as well :-)
For my personal taste I'd prefer the data dir to be hidden (think: $HOME/.ck4up) and automatically created if it doesn't exists. Rationale: as the only file the user has to create is the ck4up.conf, it doesn't seem obvious to me the she/he has to create a directory first; the DB is an internal affair and Joe User shouldn't care about it. This is just an idea though :-) (plus an attached patch)
I've included a macro @FM@ in the sample configuration file, which makes use of the freshmeat project XML database. With this, it's very lazy to add a line like 'crux md5 @FM@' to ck4up.conf and everything is done. cool!
Thanks for this nifty tool Regards Johannes -- Johannes Winkelmann mailto:jw@tks6.net Biel, Switzerland http://jw.tks6.net
Hi Johannes, hi all, On Sat, Oct 11, 2003 at 02:24:16PM +0200, Johannes Winkelmann wrote:
Hi Jürgen,
On Fri, Oct 10, 2003 at 20:09:01 +0200, Juergen Daubert wrote:
Hi all,
just in case someone is using it, attached a new version of the ck4up utility. No fancy new features, but code-cleanup and the possibility to nest "macro" definitions. Great. I'm using ck4up to check for updates and it works perfectly. I'd even suggest to create a port for this as monitoring a web site for changes is a task others probably want as well :-)
Johannes, thanks for testing and for your kind words. I'll try to put together a man-page and a port for ck4up.
For my personal taste I'd prefer the data dir to be hidden (think: $HOME/.ck4up) and automatically created if it doesn't exists. Rationale: as the only file the user has to create is the ck4up.conf, it doesn't seem obvious to me the she/he has to create a directory first; the DB is an internal affair and Joe User shouldn't care about it.
This is just an idea though :-) (plus an attached patch)
Good ideas ! Both will go into the next version, and thanks for the patch.
I've included a macro @FM@ in the sample configuration file, which makes use of the freshmeat project XML database. With this, it's very lazy to add a line like 'crux md5 @FM@' to ck4up.conf and everything is done. cool!
btw, a good starting point for a ck4up.conf, using the @FM@ macro, could be the output of something like this: prt-get printf '%n\tmd5\t@FM@\t%M\n' | grep 'Jürgen' | cut -f1,2,3 Greetings Jürgen -- juergen.daubert@t-online.de
participants (5)
-
Johannes Winkelmann -
Juergen.Daubert＠t-online.de -
juergen.daubert＠t-online.de
-
Robert McMeekin -
Simone Rota