| Pages: [1] 2 :: one page |
| Author |
Thread Statistics | Show CCP posts - 0 post(s) |

Xaen
Caldari Black Podding
|
Posted - 2008.02.24 04:25:00 -
[1]
I got really tired of EVE making a billion log files in it's log directories, so I wanted to condense them so windows could open those directories without choking on the sheer number of files.
So I wrote a couple of bash scripts that will aggregate all those files into a much smaller number of them, completely dynamically. The Gamelogs are simply appended to one aggregate file in order by date modified. So they should end up in chronological order as long as you have not manually edited any of them.
The chat logs are aggregated the same way, except individually by channel name.
If you're running Windows you can install Cygwin and then run them. If you're running *nix, you probably don't need instructions.
To "install" the scripts save the contents of each file as designated by the "author" to a scripts directory in your home directory in cygwin. Then add that directory to your $PATH.
To use them, start up your shell and cd to your C:\Documents and Settings\<username>\My Documents\EVE\logs\<logdir> directory and run the appropriate script. For Chatlogs run aggregatechatlogs.sh. For Gamelogs run aggregategamelogs.sh.
They may take several minutes to run if you have several thousand log files like I did. Be patient.
I make no warranties as to the function of these scripts. And I take no responsibilities if you somehow hose your system with them. Use at your own risk.
Originally by: aggregategamelogs.sh
#!/bin/bash
for logfile in $(ls -lt -r *_*.txt | sed -e 's/ \+/ /g' | cut -f 9 -d " "); do nice -20 cat $logfile >> Aggregate_Gamelogs.txt nice -20 rm $logfile done
Originally by: aggregatechatlogs.sh
#!/bin/bash
aggregatechattype() { type=$1 for logfile in $(ls -lt -r "$type"_* | sed -e 's/ \+/ /g' | cut -f 9 -d " "); do nice -20 cat $logfile >> Aggregate_"$type"_Log.txt nice -20 rm $logfile done }
for type in $(ls *.txt -1 | grep -v Aggregate | sed -r -e 's/(.+)_[0-9]{8,8}_[0-9]{6,6}\.txt/\1/' | sort | uniq); do echo "Aggegating $type logs" aggregatechattype $type done
-- Support fixing the EVE UI | Suggest Jita fixes |

Xaen
Caldari Black Podding
|
Posted - 2008.02.24 04:26:00 -
[2]
Reserved for the inevitable questions. -- Support fixing the EVE UI | Suggest Jita fixes |

Xaen
Caldari Black Podding
|
Posted - 2008.02.24 04:27:00 -
[3]
Reserved for the inevitable questions. -- Support fixing the EVE UI | Suggest Jita fixes |

Adunh Slavy
Ammatar Trade Syndicate
|
Posted - 2008.02.24 06:12:00 -
[4]
Could try a batch too,
for /f %%a in ('dir /b') do type %%a >>logs.new del *.txt ren logs.new logs.%random%.txt
Don't need cygwin and all that. |

Xaen
Caldari Black Podding
|
Posted - 2008.02.24 06:23:00 -
[5]
Originally by: Adunh Slavy Could try a batch too,
for /f %%a in ('dir /b') do type %%a >>logs.new del *.txt ren logs.new logs.%random%.txt
Don't need cygwin and all that.
That does not accomplish the same thing.
You get all your log files in one, new randomly named file. Subsequent executions append that new file at random to the next random one. In other words, it makes a mess of your log files.
Mine creates better organiztion.
i.e. you get something like this: Aggregate_Corp_Log.txt Aggregate_Fleet_Log.txt Aggregate_Gang_Log.txt Aggregate_Local_Log.txt Aggregate_Private_Chat_Log.txt Aggregate_Squad_Log.txt
Without ever having to specify names, or get random log file names.
Subsequent executions append new files to the existing aggregate ones in the order they were created (so logs end up in correct, chronological order). - Support fixing the UI|Suggest Jita fixes|Compact logs |

Wet Ferret
|
Posted - 2008.02.24 07:55:00 -
[6]
wtf with your sig, Xaen?
|

Adunh Slavy
Ammatar Trade Syndicate
|
Posted - 2008.02.24 07:59:00 -
[7]
True, it is not the same thing, so much as illustrating that you don't need bash. Not saying it is bad, just not necessary. The DIR argument of FOR could take globs, could use nested FOR to iterate a list of prefixes, etc.
The Real Space Initiative - V5 (Forum Link)
|

Xaen
Caldari Black Podding
|
Posted - 2008.02.24 08:17:00 -
[8]
Originally by: Wet Ferret wtf with your sig, Xaen?
Tired of waiting for them to hire a hire a UI person. - Support fixing the UI|Suggest Jita fixes|Compact logs |

Xaen
Caldari Black Podding
|
Posted - 2008.02.24 08:19:00 -
[9]
Originally by: Adunh Slavy True, it is not the same thing, so much as illustrating that you don't need bash. Not saying it is bad, just not necessary. The DIR argument of FOR could take globs, could use nested FOR to iterate a list of prefixes, etc.
You do need bash. Or some other more powerful language than batch/dos commands to accomplish what my script does.
If you want a huge mess, yes batch files will work. - Support fixing the UI|Suggest Jita fixes|Compact logs |

Adunh Slavy
Ammatar Trade Syndicate
|
Posted - 2008.02.24 10:07:00 -
[10]
Originally by: Xaen
If you want a huge mess, yes batch files will work.
With out cheating by using resource kit toys and sed compiled for windows, you win :)
The Real Space Initiative - V5 (Forum Link)
|

flashfreaking
LFC
|
Posted - 2008.02.24 10:58:00 -
[11]
your script nukes the forum, please, make linkies, or make the font smaller. Thank you Disallowed sig graphic. Send an e-mail to [email protected] when it meets the forum signature guidelines. ~Saint |

Erotic Irony
0bsession
|
Posted - 2008.02.24 14:12:00 -
[12]
Edited by: Erotic Irony on 24/02/2008 14:13:07
Originally by: Wet Ferret wtf with your sig, Xaen?
Xaen, have you thought about making your sig a jpg of one of eve's modal windows,
"Support UI Repair" [Ok]
edit: I've been logged out twice while trying to post this, surprise surprise.
___ Eve Players are not very smart. Support Killmail Overhaul
|

Xaen
Caldari Black Podding
|
Posted - 2008.02.24 17:35:00 -
[13]
Originally by: flashfreaking your script nukes the forum, please, make linkies, or make the font smaller. Thank you
I can't make the font smaller. That's what the code tag gives you.
Links over complicate things. - Support fixing the UI|Suggest Jita fixes|Compact logs |

Xaen
Caldari Black Podding
|
Posted - 2008.02.24 17:36:00 -
[14]
Originally by: Erotic Irony Edited by: Erotic Irony on 24/02/2008 14:13:07
Originally by: Wet Ferret wtf with your sig, Xaen?
Xaen, have you thought about making your sig a jpg of one of eve's modal windows,
"Support UI Repair" [Ok]
edit: I've been logged out twice while trying to post this, surprise surprise.
Sadly, I don't think anyone would get it. - Support fixing the UI|Suggest Jita fixes|Compact logs |

Vanora Kime
Caldari Reapage Inc.
|
Posted - 2008.02.25 15:37:00 -
[15]
I like your idea, very useful. I actually didn't even dare to open one of the log directories, as it would have taken hours to just show me the files ... I put both your scripts in a single one and made it a bit shorter, the result is the same:
Quote:
#!/bin/bash
# Gamelogs: GAMELOG_DIR="/mnt/sata2/Eigene Dateien/eve/logs/Gamelogs" cd "$GAMELOG_DIR" && find . -name "*.txt" -exec cat "{}" >> ./gamelogs.log ";" -exec rm "{}" ";"
# Chatlogs, one file for each chat window: CHATLOG_DIR="/mnt/sata2/Eigene Dateien/eve/logs/Chatlogs" cd "$CHATLOG_DIR" && for type in $(ls *.txt | sed -r -e 's/(.+)_[0-9]{8,8}_[0-9]{6,6}\.txt/\1/' $ find . -name "$type*.txt" -exec cat "{}" >> "$type.log" ";" -exec rm "{}" ";" done
I've put this into a cronjob that runs every night, no more billions of logfiles for me, yay! Thanks for this one :-)
|

Garat Mant
Minmatar Moons of Pluto
|
Posted - 2008.02.25 15:51:00 -
[16]
At work, so no access to logfiles - do chatlogs have your play start and end timestamps in them? i.e. with some careful parsing, could you get aggregate hours played by week/month? --
|

An Anarchyyt
Gallente Battlestars GoonSwarm
|
Posted - 2008.02.25 15:53:00 -
[17]
Horizontal scroll is much worse than any logfiles.
Also, I hope this counts as modding Eve and you get banned.
Originally by: CCP Wrangler Second, a gentile is a non jewish person
|

Lil'Red Ridin'Hood
Snake Assault
|
Posted - 2008.02.25 16:05:00 -
[18]
When I get home, I'll build a small executable doing the same. I hope you don't mind if I copy your idea for that.
No need for cygwin then. Select log-folder, click run, cross your fingers, done.
|

Xaen
Caldari Black Podding
|
Posted - 2008.02.25 17:02:00 -
[19]
Originally by: Vanora Kime I like your idea, very useful. I actually didn't even dare to open one of the log directories, as it would have taken hours to just show me the files ... I put both your scripts in a single one and made it a bit shorter, the result is the same:
I've put this into a cronjob that runs every night, no more billions of logfiles for me, yay! Thanks for this one :-)
You're welcome.
But careful running find. The scripts I wrote sort by modification date so that the files are appended in chronological order. - Support fixing the UI|Suggest Jita fixes|Compact logs |

Xaen
Caldari Black Podding
|
Posted - 2008.02.25 17:15:00 -
[20]
Originally by: Garat Mant At work, so no access to logfiles - do chatlogs have your play start and end timestamps in them? i.e. with some careful parsing, could you get aggregate hours played by week/month?
Yes they do.
But I hear using the logserver is a more effective way to get time played. - Support fixing the UI|Suggest Jita fixes|Compact logs |

Xaen
Caldari Black Podding
|
Posted - 2008.02.25 17:15:00 -
[21]
Originally by: An Anarchyyt Horizontal scroll is much worse than any logfiles.
Also, I hope this counts as modding Eve and you get banned.
You've got to be kidding me. - Support fixing the UI|Suggest Jita fixes|Compact logs |

Kazuo Ishiguro
House of Marbles Zzz
|
Posted - 2008.02.25 17:19:00 -
[22]
If you leave the log window open ingame, gamelogs get lumped into one file until you close it; you don't get a fresh log after each session change. I've occasionally found this useful.
My research services Spreadsheets: Top speed calculation - Halo Implant stats |

Lil'Red Ridin'Hood
Snake Assault
|
Posted - 2008.02.27 22:26:00 -
[23]
Originally by: Lil'Red Ridin'Hood When I get home, I'll build a small executable doing the same. I hope you don't mind if I copy your idea for that.
No need for cygwin then. Select log-folder, click run, cross your fingers, done.
I haven't forgotten about this, only had RL get in the way a bit. I'll do my best to have a version ready by the end of the weekend. I'd hope to have one ready before that, actually.
|

AndrewRyan
|
Posted - 2008.02.27 22:28:00 -
[24]
Bishy Bashi boshi!
Originally by: Malcanis
Seriously, have you ever actually pressed F1 with something other than an asteroid locked?
|

Scilent Enigma
Minmatar Vae Victis Inc.
|
Posted - 2008.02.27 22:41:00 -
[25]
I dunno.. I've only found one batch script useful for the log files as to date, and that is:
Quote:
del /Q "logs\Chatlogs\*.*" del /Q "logs\Gamelogs\*.*" del /Q "logs\Marketlogs\*.*"
Placed in a bat file in the EVE directory in My Documents...
Though might give cygwin a try, sounds fun being able to scrip bash in windows...
|

Maglorre
|
Posted - 2008.02.28 04:59:00 -
[26]
Edited by: Maglorre on 28/02/2008 05:00:40 Nice idea... couple of minor nit-picky things with the scripts people have posted.
[stupid bit removed, I misread the initial script]
None of the suggested scripts provide proper error testing before deleting the original files. If your append command fails for some reason then the original file is still trashed.
|

Xaen
Caldari Target Practice Inc.
|
Posted - 2008.02.28 22:21:00 -
[27]
Originally by: Maglorre None of the suggested scripts provide proper error testing before deleting the original files.
The list of files to delete is generated based on files that exist. The only way this could be a problem is if you start the script the delete one of the log files while it's aggregating that type, but before it gets to that file. And then it will just throw an error and do the rest of them.
Originally by: Maglorre If your append command fails for some reason then the original file is still trashed.
True, but you'd have to run out of disk space or some other catastrophic problem. And it's not very likely since it deletes them as it goes. The resulted aggregated files should take up the same amount of space on disk. So for in order for it to fail you'd have to have one enormous file be the first one aggregated and very little disk space left. - Support fixing the UI|Suggest Jita fixes|Compact logs |

Xaen
Caldari Target Practice Inc.
|
Posted - 2008.02.28 22:25:00 -
[28]
Originally by: Scilent Enigma I dunno.. I've only found one batch script useful for the log files as to date, and that is:
Quote:
del /Q "logs\Chatlogs\*.*" del /Q "logs\Gamelogs\*.*" del /Q "logs\Marketlogs\*.*"
Placed in a bat file in the EVE directory in My Documents...
Though might give cygwin a try, sounds fun being able to scrip bash in windows...
After working in *nix so long I can't live in windows without Cygwin. I use it almost daily.
Some asshat uploads 50 video files in 50 individual .rar files? (mp3s/vidoes rarely compress appreciably) nice -20 for x in $(ls *.rar); do rar -x "$x"; done & And walk off to the living room to stream the first one to XBMC. ;) - Support fixing the UI|Suggest Jita fixes|Compact logs |

Scilent Enigma
Minmatar Vae Victis Inc.
|
Posted - 2008.02.29 00:11:00 -
[29]
Originally by: Xaen After working in *nix so long I can't live in windows without Cygwin. I use it almost daily.
Some asshat uploads 50 video files in 50 individual .rar files? (mp3s/vidoes rarely compress appreciably) nice -20 for x in $(ls *.rar); do rar -x "$x"; done & And walk off to the living room to stream the first one to XBMC. ;)
Ahh.. the joys of *nix..
I really should learn it better, just wished I wasn't so stuck with all the programs in windows. Maybe in my new computer I will be able to install it and VMware Vista when needed..
|

Maglorre
|
Posted - 2008.02.29 01:45:00 -
[30]
Originally by: Xaen The list of files to delete is generated based on files that exist. The only way this could be a problem is if you start the script the delete one of the log files while it's aggregating that type, but before it gets to that file. And then it will just throw an error and do the rest of them.
Yeah, that's not an issue.
Originally by: Xaen
Originally by: Maglorre If your append command fails for some reason then the original file is still trashed.
True, but you'd have to run out of disk space or some other catastrophic problem. And it's not very likely since it deletes them as it goes. The resulted aggregated files should take up the same amount of space on disk. So for in order for it to fail you'd have to have one enormous file be the first one aggregated and very little disk space left.
Well, you are correct that it's not likely but that's no excuse to not implement correct error handling .
Just because you can't think of any likely modes of failure doesn't mean there is none. I try to keep this in mind whenever I'm writing code and it has saved my bacon on more than one occasion.
Take this as an example. Modify your script so that the aggregated logs are stored on a remote machine. Start the script then turn off your router.
Cheers
|
| |
|
| Pages: [1] 2 :: one page |
| First page | Previous page | Next page | Last page |