ArcEmu: [Application Release] SwiftAnalyzer - Search, sort, split and export GM log files - ArcEmu

Jump to content

Toggle shoutbox Lastest Announcements

dfighter  : (07 December 2014 - 12:06 PM) Arcemu is in hibernation mode, please read http://arcemu.org/fo...showtopic=26903
dfighter  : (01 January 2013 - 05:56 PM) Arcemu wishes you all a happy new year!
Hasbro  : (12 September 2012 - 10:01 AM) Please excuse our outage from the web! Our web host had a major malfunction!
dfighter  : (01 September 2012 - 04:05 PM) Since the spam bots just don't want to stop, I've enabled admin verification when registering.
dfighter  : (23 January 2012 - 09:56 PM) Please note that from now on you will need to confirm your email on the wiki in order to edit it!
Hasbro  : (31 December 2011 - 12:50 PM) Happy New Years all!
Navid  : (26 December 2011 - 04:09 AM) Merry Christmas !!!!!! Happy holidays all :)
WAmadeus  : (24 December 2011 - 03:54 PM) Merry Christmas to all!
dfighter  : (24 December 2011 - 11:05 AM) The Arcemu team wishes y'all a Merry Christmukkah!
Hasbro  : (05 October 2011 - 12:53 PM) Looking for web designers for upcoming web related project. If you're interested in designing user interfaces contact me
dfighter  : (02 September 2011 - 03:47 PM) So who here wants vehicles in Arcemu? :P http://arcemu.org/fo...showtopic=25440
Hasbro  : (14 August 2011 - 03:25 PM) Join us on irc, grab an irc client and connect to irc.freenode.net join channel #arcemu /server irc.freenode.net:6667 /join #arcemu
jackpoz  : (03 August 2011 - 05:33 AM) to all Lua Engine (old one) users: please check http://arcemu.org/fo...showtopic=25274
Hasbro  : (20 May 2011 - 05:27 PM) Looking for people experienced with CMake configuration and setup! Contact me asap
Hasbro  : (15 May 2011 - 05:03 PM) ArcEmu is recruiting C++ programmers, contact Hasbro if interested.
paroxysm  : (03 May 2011 - 06:26 PM) Updated luabridge gossip example to describe the whole gossip creation process rather than just how to create menu. Gossip tutorial
paroxysm  : (23 April 2011 - 11:35 AM) Lua writers can refer to the Luabridge Tutorials section in the Wiki to learn how to write gossip code correctly.
Hasbro  : (20 April 2011 - 05:22 PM) Thank you for your continuous contribution of bug reports, we are working on them.
Hasbro  : (17 April 2011 - 03:20 AM) Please consider donating to support our bills. Donations can be sent using PayPal to donations@arcemu.org - Thank you for your support.
paroxysm  : (10 April 2011 - 12:43 AM) Refer to the Luabridge Tutorials section in the Wiki to learn the new syntax of luabridge.
Resize Shouts Area

Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

[Application Release] SwiftAnalyzer - Search, sort, split and export GM log files Parses GM command log files and allows you to search, sort, export...

#1 User is offline   Twist3d 

  • Member
  • Pip
  • Group: Members
  • Posts: 38
  • Joined: 29-July 08

  Posted 12 February 2009 - 11:59 PM

SwiftAnalyzer 1.0.3332.26827 BETA
GM Log Parser for Ascent-based servers.

This application is still a work in progress!!!

What is SwiftAnalyzer?
SwiftAnalyzer is an application for Windows that makes it much easier to work with the gm command logfiles that Ascent generates. With SwiftAnalyzer, you can select the logfile you want to read, and the program will parse every line in the file, organizing all the data into neat, easy-to-read columns. You can sort the data by any column, and more importantly, you can search and filter the data so you see only the records you are looking for.


How does it work, and how do I use it?
SwiftAnalyzer works by opening the logfile you specify and reading every line in the file. As a line is read, a regex is applied to the line to extract important data from it such as date/time, account id, account name, player name, and the action performed. This data is then inserted into a table under its respective columns.
Once this process completes, which usually takes about one second per 5,000 lines of data, the Logfile Analyzer window will open with the table data filled in. You can then sort the data by clicking on a column header, and you can filter the table to show only the records you want by using the Filter Designer.
Say you want to find all the records of any time the GM Twisted used the kill command: you simply add a filter for "Twisted" in the "Player Name" column, and another filter for "used kill" in the "Action" column. Then click apply filter, and you'll see every time Twisted used .kill. You can add as many filters as you want, and you can also fine tune the comparison type between "contains", "doesn't contain", "equals", and "doesn't equal".
Filters are for the most part applied instantly except when using the <All fields> filter, which searches in every column, in which case the amount of time for the filter to complete depends on how many rows you have in your log. You generally should avoid using the All fields filter if your log has more than 75-100k rows.

While SwiftAnaylzer can technically handle logs of any size, it works best with logfiles under 150-200k rows. The more rows your file has, the longer it takes SwiftAnalyzer to parse the file, sort it, and filter it. For this reason, SwiftAnalyzer also includes a "Split File" function, which will split your gmcommand.log file once it gets to be too big. [this feature is currently NYI]

In addition to all of the above, SwiftAnalyzer can also export data to CSV (comma separated value) file for use in programs like Microsoft Excel, OO.org Calc, etc. as well as an HTML file in case you want a formatted version of the logfile. Why you'd need this, I don't know, but I added it out of boredom anyway. [this feature is currently NYI]

As mentioned above, this application is still somewhat in development as I'm working on adding the few missing features. I am currently aware of the following issues that need to be fixed:
-Split does not work
-Settings does not work
-Export does not work

If you find an issue not mentioned above, please report it to me via email (twist3df4te at gmail dot com), PM, or a post here.

SwiftAnalyzer was written in VC# 2005; I may consider releasing the code once everything's finished and I've cleaned up/commented it.


Screenshots
Posted Image
Posted Image


Download
v1.0.3332.26827: http://www.leetsoft....iftAnalyzer.exe

Scan results (http://virusscan.jotti.org):

Quote

File: SwiftAnalyzer.exe v1.0.3332.478
Status: OK
MD5: 13378880b44ed05a8d0dd50a166c12b2 (lol md5 starts with 1337)
Packers detected: -
Scan taken on 14 Feb 2009 16:03:41 (GMT)
A-Squared
Found nothing
AntiVir
Found nothing
ArcaVir
Found nothing
Avast
Found nothing
AVG Antivirus
Found nothing
BitDefender
Found nothing
ClamAV
Found nothing
CPsecure
Found nothing
Dr.Web
Found nothing
F-Prot Antivirus
Found nothing
F-Secure Anti-Virus
Found nothing
G DATA
Found nothing
Ikarus
Found nothing
Kaspersky Anti-Virus
Found nothing
NOD32
Found nothing
Norman Virus Control
Found nothing
Panda Antivirus
Found nothing
Sophos Antivirus
Found nothing
VirusBuster
Found nothing
VBA32
Found nothing


Enjoy, please post comments.

0

#2 User is offline   Hoffa 

  • Sup dawg
  • Group: Retired
  • Posts: 1,159
  • Joined: 11-December 08
  • Gender:Male
  • Location:Sweden
  • Interests:Trolling

Posted 13 February 2009 - 01:41 AM

Nice :D
Posted ImagePosted Image



This person is known for trolling, flaming and extreme ignorance. Respond to at own risk.
0

#3 User is offline   FakeNinja 

  • Member
  • Pip
  • Group: Members
  • Posts: 12
  • Joined: 26-January 09

Posted 13 February 2009 - 12:02 PM

Cool program!
0

#4 User is offline   MesoX 

  • Advanced Member
  • PipPipPip
  • Group: Members
  • Posts: 296
  • Joined: 15-June 08
  • Gender:Male
  • Location:Czech Republic
  • Interests:pc & girls & parties

Posted 13 February 2009 - 02:11 PM

Aaah thanks for it! Very nice man :huh:
Posted Image
0

#5 User is offline   friskey4u 

  • Enthusiast
  • PipPipPip
  • Group: Members
  • Posts: 310
  • Joined: 19-December 08
  • Gender:Male
  • Location:U.S.A
  • Interests:Music

    Skip school, fly kites.

Posted 13 February 2009 - 05:08 PM

Could you give us a better description as to what it does?

I sort of understand..
0

#6 User is offline   Twist3d 

  • Member
  • Pip
  • Group: Members
  • Posts: 38
  • Joined: 29-July 08

Posted 13 February 2009 - 05:18 PM

Sorry I was busy when I posted the original post and I didn't have much time to explain.

As you may already know, all flavors of Ascent have the ability to output a log file of commands used by GMs on your server. This log file is stored in \logs\gmcommand.log. It will show the date/time, account ID, account name, character name, and the command they used (for example "used kill command on CREATURE Rat" or "spawned gameobject sqlid 123456 at x,y,z,map, saved to DB"). The purpose of the GM log is for abuse investigation in case you've got someone abusing their powers.

Normally you can browse through this file with notepad, but it's a pain in the ass since notepad is ugly and there is no way to sort or filter.

SwiftAnalyzer opens the gmcommand.log file and reads each line. Then it uses a regex to parse the important information from each line and it organizes it into neat columns. You can sort the list by any column you'd like (such as date or character name), and more importantly you can filter which lines it shows, for example, only show commands used by account JONNY on February 3.

The application also (soon) allows you to split a large GM log file into multiple parts to make them easier and faster to work with. In addition you'll soon be able to export either the entire log or just parts of it to an HTML file or a CSV file. The uses for this may not be immediately obvious but it's fairly trivial for me to add and it might help someone.

Pretty soon I hope to take the parsing part further and let it parse names, ids, etc. from the "action" field (for example, it can get the sqlid from the gameobject spawn command. That's going to take a while for me to figure out how to display the data so it'll come last.

Anyway if you have suggestions please post them, hope this cleared up what the addon does.
0

#7 User is offline   alleycat 

  • Helps so much & is so mature & gay
  • PipPipPipPipPip
  • Group: Members
  • Posts: 835
  • Joined: 25-July 08
  • Gender:Male

Posted 13 February 2009 - 11:17 PM

Sweet! I suck at parsing so this is great :-).
I can haz cookie?
0

#8 User is offline   Twist3d 

  • Member
  • Pip
  • Group: Members
  • Posts: 38
  • Joined: 29-July 08

Posted 14 February 2009 - 12:22 AM

Updated it right now, SwAn now includes:
-Autoupdater
-All fields filter option now works
-Can now open files that are in use by ascent (makes a temp copy first, then parses that one)
-Filtering is MUCH more robust, you can add as many filters as you want and apply them all at the same time (there's a checkbox next to the filter that's supposed to enable/disable it, it doesn't work at the moment, I'll add it next update)
-Added/replaced some icons
-Added unhandled exception handling (heh.)

Thanks to everyone who is downloading. ^_^
0

#9 User is offline   Dr.Overclock 

  • Enthusiast
  • PipPipPip
  • Group: Members
  • Posts: 243
  • Joined: 05-September 08
  • Location:Italy, Milan
  • Interests:ArcEMU<br />C++<br />MySQL<br />PHP<br />Emulation

  Posted 14 February 2009 - 01:26 AM

Thanks you, really usefull.

I speak a bit English/Io parlo Italiano molto bene/Русский - мой родной язык.
Io parlo un po Inglese/Я говорю на Итальянском очень хорошо/Russian is my native language.
Я говорю немного на Английском/I speak Italian very well/Russo e' mio madre-lingua.
0

#10 User is offline   HalestormXV 

  • The Ultimatum
  • PipPipPipPipPipPip
  • Group: Members
  • Posts: 1,031
  • Joined: 07-June 08
  • Gender:Male

Posted 18 February 2009 - 01:18 PM

You know what. I like this program alot and it seems to be lively and a very good favorite of people so. I am going to sticky it. we haven't had a sticky tool in a good while and this certainly deserves it. Keep t he development up it looks very promising and already works wonders.
**Owner of the Serenade of Sorrow Funserver**
My Most Recent Video: The Book of Kidou
Posted Image
0

#11 User is offline   Twist3d 

  • Member
  • Pip
  • Group: Members
  • Posts: 38
  • Joined: 29-July 08

Posted 19 February 2009 - 04:36 PM

View PostHalestormXV, on Feb 18 2009, 01:18 PM, said:

You know what. I like this program alot and it seems to be lively and a very good favorite of people so. I am going to sticky it. we haven't had a sticky tool in a good while and this certainly deserves it. Keep t he development up it looks very promising and already works wonders.


Yay, second sticky! ^_^

Thanks for sticky-ing it, I've been taking a break from this app to work on SwiftGM but hopefully I can get Splitting and Exporting working soon.
0

#12 User is offline   Lawyer 

  • ArcEmu Lover
  • Group: Banned
  • Posts: 412
  • Joined: 06-July 08
  • Gender:Male

Posted 20 February 2009 - 08:44 AM

View PostTwist3d, on Feb 19 2009, 09:36 PM, said:

Yay, second sticky! :)

Thanks for sticky-ing it, I've been taking a break from this app to work on SwiftGM but hopefully I can get Splitting and Exporting working soon.


Just wondering, Twist3d

Could you llink this to your "Swift Discipline" So admin's can look up the Log's ingame?

Would be good ?

Just a suggestion


~Lawyer
Super Moderator On - Squire-DB

Can you beat me? - http://ogchaos.mybrute.com

#13 User is offline   Twist3d 

  • Member
  • Pip
  • Group: Members
  • Posts: 38
  • Joined: 29-July 08

Posted 20 February 2009 - 03:15 PM

View PostLawyer, on Feb 20 2009, 08:44 AM, said:

Just wondering, Twist3d

Could you llink this to your "Swift Discipline" So admin's can look up the Log's ingame?

Would be good ?

Just a suggestion


~Lawyer


Unfortunately this is not really feasible due to two rather large caveats:
Firstly, addons cannot directly interact with the "outside world," that is, they cannot do anything outside of WoW including file I/O. This is the limiting factor here, without this I cannot read from the logfile. It might be possible to do some advanced memory hacking or something to inject values into a Lua variable but that is far, far beyond the scope of what I can do, if it is even possible.

And of course the second problem is that the logfile is stored on the server, not on the connecting client's computer, so data would have to be transmitted to the client on the fly. Not particularly difficult but again with the limitation of WoW's addon system there is not much I can do.

Thanks for the suggestion anyway :P
0

#14 User is offline   MegaBigBoss 

  • Member
  • Pip
  • Group: Members
  • Posts: 23
  • Joined: 20-September 08

Posted 27 February 2009 - 04:50 PM

Thanks for this, but you should make it like this :

A server, which load gmfile every 2 or 5 minutes, who wait for client.
And a little client, who can send request of search. Like this there is so much problems who disappear, like "How can I do with a fuckin' 400mb logfile ?".
It's just a sugestion, but I think it would be greater :lol:
0

#15 User is offline   Twist3d 

  • Member
  • Pip
  • Group: Members
  • Posts: 38
  • Joined: 29-July 08

Posted 28 February 2009 - 01:12 PM

View PostMegaBigBoss, on Feb 27 2009, 04:50 PM, said:

Thanks for this, but you should make it like this :

A server, which load gmfile every 2 or 5 minutes, who wait for client.
And a little client, who can send request of search. Like this there is so much problems who disappear, like "How can I do with a fuckin' 400mb logfile ?".
It's just a sugestion, but I think it would be greater :P


Huh... this is actually a pretty good idea, I like it!

I'll see if I can get this added, especially since I haven't made any projects with p2p communication in C# yet, so it should definitely be an experience.

However there are definitely going to be some challanges. Some of the ones I can immediately see:
-Memory usage on the server. Having the application keep a 50mb logfile in memory (you said 400mb lol, I do NOT recommend using anything that big as it will probably be several million records, which will very likely kill SwiftAnalyzer), while the server is at the same time dealing with ascent, may not be a great idea for popular and/or memory-limited servers.
-CPU usage when reloading every 4-5 minutes. I haven't tested but I can't imagine that running half a million regexes would be fast or efficient, and it may cause some lag for players, especially since it would occur every 5 minutes. Maybe a 30-minute update delay would be good, or maybe have the client request that the server update its "cache".
-Serving search results. For searches that return many results there may have to be some pagination. In addition there may be a significant delay between the time it takes for the client to actually request a search result and the time it is delivered, since the filter has to be applied first, and then the results have to be sent (probably line by line) over the internet. For this reason I'm thinking there will be paging going on every 100 lines.
-Multithreading. I'm going to have to use lots of backgroundworkers for this to make sure the server app does not freeze/lockup while processing/sending data or responding to requests. I would use regular old code-only thread objects except for the fact that they are a pain in the ass to work with.
-Authentication. You don't want to have this data accessible to just anyone, and I might work in encryption just for fun as an optional thing.

And there will likely be more. However I want to try this out simply because it sounds like it could be fun to code.

Thanks for the idea! :D
0

#16 User is offline   MegaBigBoss 

  • Member
  • Pip
  • Group: Members
  • Posts: 23
  • Joined: 20-September 08

Posted 04 March 2009 - 08:29 AM

View PostTwist3d, on Feb 28 2009, 07:12 PM, said:

-Memory usage on the server. Having the application keep a 50mb logfile in memory (you said 400mb lol, I do NOT recommend using anything that big as it will probably be several million records, which will very likely kill SwiftAnalyzer), while the server is at the same time dealing with ascent, may not be a great idea for popular and/or memory-limited servers.


Hmm, for this one, a splitter could be good, or an optional conf in ascent which active or not the splitter, and if activated ascent will make a folder "SplitLog" intlo log folder, and where it would be things like that :
GameObject Commands
NPC Commands
Ban commands

etc ...
and by hour like it :
SplitLog
--21-02-09.18h00
----Gmcommand.log
----Thing.log
etc ..
It should make it easier (i think)


Quote

CPU usage when reloading every 4-5 minutes. I haven't tested but I can't imagine that running half a million regexes would be fast or efficient, and it may cause some lag for players, especially since it would occur every 5 minutes. Maybe a 30-minute update delay would be good, or maybe have the client request that the server update its "cache".


Hmm, yeah, but with the split idea, it could be like it :
client send request, if the file has been changed, the server reload it.
So it would be more easy.


Quote

I'll see if I can get this added, especially since I haven't made any projects with p2p communication in C#


Oh :P :P :( :( :( :( !!!!
C# isn't linuxable ! :(
(I would prefer it in C :P but it's your choice =p)
0

#17 User is offline   MaToX 

  • Member
  • Pip
  • Group: Members
  • Posts: 17
  • Joined: 08-June 08

Posted 19 April 2009 - 03:04 PM

Hey man i want to know how i need to do for when i muted 1 accound and ban 1 account appear 1 announce like GM: player was muted or banned from the server by GM, reason:

please tell me i want to modify it please
0

#18 User is offline   darkwolf007 

  • Member
  • Pip
  • Group: Members
  • Posts: 48
  • Joined: 10-January 09

Posted 20 April 2009 - 02:12 PM

View PostMaToX, on Apr 19 2009, 08:04 PM, said:

Hey man i want to know how i need to do for when i muted 1 accound and ban 1 account appear 1 announce like GM: player was muted or banned from the server by GM, reason:

please tell me i want to modify it please

try saying that sentence correctly so we can understand you and this is a dead post dont bring up dead posts
0

Share this topic:


Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users