Pages: 1 2 3 4 [5] 6 7 8 9 10 11 12 13 14 15 16 17 18 .. 18 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 1 post(s) |
PsyKzz
Bat Country Goonswarm Federation
7
|
Posted - 2011.11.20 14:45:00 -
[121] - Quote
I was wondering if you could explain a way i could link data back to the API Key it came from?
I know you make use of ownerID for alot of data, but can you add a way to trace it back to the api key? Meh. |
Dragonaire
Corax. PURgE Alliance
19
|
Posted - 2011.11.20 23:37:00 -
[122] - Quote
The only APIs where there is a direct link from the key to ownerID is in account section. The eve, map, and server sections don't really have an owner so they default to 0. Char section APIs use the characterID and corp section uses the corporationID but since Yapeal can work with multiple keys for each there's really no direct link to the key used. To put it another way Yapeal doesn't care if there are one or multiple keys active for the same character or corporation and will use them in a random order when trying to retrieve the XML data from the API servers so there really isn't a way to say which key will be used.
If you enforce only one key of a type on a per character/corporation basis in your application you can then try to do a one to one mapping but watch out for account type keys as you actually have a one to many relationship with them.
Hope that helps you. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
PsyKzz
Bat Country Goonswarm Federation
7
|
Posted - 2011.11.21 10:24:00 -
[123] - Quote
I guess for the nature of my application i think i should just maintain another table that lists the character / corporation IDs it has access to, that way i can just counter reference the table to find the key i need.
Meh. |
Dragonaire
Corax. PURgE Alliance
19
|
Posted - 2011.11.21 10:32:00 -
[124] - Quote
Or maybe learn to use the accountApiKeyInfo, accountCharacters, and accountKeyBridge which really does the linking you were asking about there just isn't a real one to one link except as I said you enforce one in your application. You can do that with a few SQL queries on the exist info from the above tables. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Sable Blitzmann
Massively Dynamic The Jagged Alliance
15
|
Posted - 2011.11.21 15:26:00 -
[125] - Quote
Is there any way to delete all records associated with an API? |
Bado Roul
Deep Core Mining Inc. Caldari State
0
|
Posted - 2011.11.21 22:58:00 -
[126] - Quote
I updated Yapeal last Saturday (zip version) and I'm having high CPU usage since Sunday downtime. I see all log files empty with default Yapeal config. How can I get logging to work?
Thanks! |
Dragonaire
Corax. PURgE Alliance
19
|
Posted - 2011.11.21 22:58:00 -
[127] - Quote
you can always truncate the tables and as long as you don't have that API active it won't refill. Yapeal actually does that on a few of the tables in eve, map etc so it can use inserts instead of upsert which is slower. If it's a table in account, char, or corp sections then you'll need to use the correct ownerID to do so. Here again at times Yapeal uses delete with ownerID to clear stuff out before inserting the new data with an example being assetList where upserts don't really work.
Now if the question is there something in Yapeal to help you do this? the answer is no it just does this things as need during it's own work updating APIs. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Dragonaire
Corax. PURgE Alliance
19
|
Posted - 2011.11.21 23:00:00 -
[128] - Quote
@Bado Roul look in the config/yapeal.ini file for the cache settings you probably need to change. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Bado Roul
Deep Core Mining Inc. Caldari State
0
|
Posted - 2011.11.21 23:24:00 -
[129] - Quote
Dragonaire wrote:@Bado Roul look in the config/yapeal.ini file for the cache settings you probably need to change.
I have all default values, except MySQL user and password, so it's "file".
I disabled all my characters inside MySQL table and I got an INFO message in stdout and inside log/yapeal_info.log correctly (INFO: No active characters for char section). What I need is some kind of "debug" level and I don't know how to get it. I've tried with "level value="debug"" (logger.xml) and still 0 bytes. |
Dragonaire
Corax. PURgE Alliance
20
|
Posted - 2011.11.22 05:01:00 -
[130] - Quote
I haven't had a chance to really add any extra debug stuff to Yapeal since the change to new error logging code but you can try trace but it'll probably not have much interesting either but at least you'll know what all code it does run . Have you run install/checkForRequirements.php yet it might spot something that should help like the cache directory not being writable etc. Clear up any errors it might find and see if that helps. If doing those things doesn't seem to fix it let me know and I'll see what else I can come up with that might give us a clue to why it's not caching anything. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
|
Bado Roul
Deep Core Mining Inc. Caldari State
0
|
Posted - 2011.11.22 17:17:00 -
[131] - Quote
Trace isn't useful, I got about 130MB log file (I increased value in logger.xml) which I won't read :P
I've notice some happend to my activeAPIMask in MySQL because isn't the same as my EVE API key. corpWalletJournal was outdated (since october 23rd) and corpWalletTransactions since november 13th.
I'm now running cronjob every hour to fully update those tables since walking isn't adding all the records up to now.
Now it seems it takes 7 minutes and my RRDtool graph shows 70% CPU usage spike (before saturday, it was less than 10% CPU usage)
I don't know if the next quote is useful to check if this is normal behaviour (API access and cached files):
Quote:2011/11/22 04:43:06 x.x.x.x /corp/AccountBalance.xml.aspx ? xxxxxxxx OK 2011/11/22 04:43:05 x.x.x.x /corp/Contracts.xml.aspx OK 2011/11/22 04:43:05 x.x.x.x /corp/WalletJournal.xml.aspx OK 2011/11/22 04:43:05 x.x.x.x /corp/WalletJournal.xml.aspx OK 2011/11/22 04:43:05 x.x.x.x /corp/WalletJournal.xml.aspx OK 2011/11/22 04:36:13 x.x.x.x /corp/WalletJournal.xml.aspx OK 2011/11/22 04:36:03 x.x.x.x /corp/WalletJournal.xml.aspx OK 2011/11/22 04:36:03 x.x.x.x /corp/WalletJournal.xml.aspx OK 2011/11/22 04:36:03 x.x.x.x /corp/WalletJournal.xml.aspx OK
-rw-r--r-- 1 bado bado 348 Nov 22 17:36 WalletJournaldf89d9d2fb5a0c1de2c6de44c2a6e1030793a86a.xml -rw-r--r-- 1 bado bado 348 Nov 22 17:36 WalletJournaldbd729fc48a67d5d4e0cd93c81e9f1b5abf56546.xml -rw-r--r-- 1 bado bado 348 Nov 22 17:36 WalletJournal849701b6c72ecdaac99ff15a1816dfd502d910e5.xml -rw-r--r-- 1 bado bado 223922 Nov 22 17:36 WalletJournalec3a1cee7d1729e20561abb49440160016c1ae0f.xml -rw-r--r-- 1 bado bado 348 Nov 22 17:43 WalletJournale4c54bc732801229ae0aadd5e75567d90066973f.xml -rw-r--r-- 1 bado bado 348 Nov 22 17:43 WalletJournal858a2027a973ef6e13c6338a235629b4856909f9.xml -rw-r--r-- 1 bado bado 348 Nov 22 17:43 WalletJournal678cb70b57d8db1e93aff6de1c327445427a7035.xml -rw-r--r-- 1 bado bado 1406 Nov 22 17:43 Contracts11c39b5acd8164349d5561246e6ced567329e803.xml -rw-r--r-- 1 bado bado 788 Nov 22 17:43 AccountBalancef888b9df2c746a5005b54a23baf3f4a7c8e27a0e.xml
I will run cronjob untill all records are up to date to see if CPU usage decrease.
Thanks! |
Dragonaire
Corax. PURgE Alliance
20
|
Posted - 2011.11.22 18:12:00 -
[132] - Quote
I'll make a bet if you looked at what is using mast of your CPU it will turn out to be MySQL trying to put everything into the tables. Also it really is better to run Yapeal every minute as then it doesn't get behind and have to run for extended periods to catch up all the time. It has some built-in code to try evening out globs of APIs needing retrieved all at the same time that can't do it's job if ran infrequently. Everyone is always shooting themselves in the foot trying to out think me and Yapeal on that
It will be very busy for a while when you first start it as it does have to try getting all of the APIs for everything you have asked it to do but once that initial fill is done it should within a couple days only have an occasional spike when a couple of the larger APIs happen to need to be done at the same time.
Yapeal itself spends 99% of it's time waiting on either the API servers to give it the data it needs from the network or waiting on MySQL to finish storing the data in the database the code itself usually only takes a second or two at most to run and typically only is doing something for 1/4 of a second or less of the time it take to do an API. Disk I/O has been the most common limitation on how many keys it can handle on most shared hosts as they are usually setup for lots of reads but few writes to the file system but Yapeal spends 90% of it's time writing directly for cache or indirectly through database table inserts in MySQL.
Hopefully that helps you understand what's going on. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Bado Roul
Deep Core Mining Inc. Caldari State
0
|
Posted - 2011.11.22 19:07:00 -
[133] - Quote
Thanks for explanation, I will run Yapeal more often then
I did many "ps" during Yapeal execution and this is common usage: (crontab started at 19:36)
Quote:root:~# date && ps aux | egrep -E -i "/usr/bin/php | /usr/sbin/mysqld" | grep -v egrep Tue Nov 22 19:43:25 CET 2011 mysql 863 0.6 11.3 164708 58860 ? Sl Nov21 17:18 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 bado 23004 86.6 3.9 69068 20644 ? S 19:36 6:23 /usr/bin/php -Cq /home/bado/doc/yapeal/yapeal.php
I don't use shared hosting, I have Linode 512: http://dl.dropbox.com/u/72781/eve/eveforo/cpu.PNG http://dl.dropbox.com/u/72781/eve/eveforo/io.PNG
As I said before, I was running Yapeal smoothly for over 2 months with 2h cronjob. This problem is very recent.
I hope when all data is filled (those corpWallet* outdated), CPU usage go down again
Thanks! |
Norian Lonark
Black Thorne Corporation Black Thorne Alliance
2
|
Posted - 2011.11.25 09:41:00 -
[134] - Quote
Hi Everyone,
I have a bit of an odd problem. Yapeal has been working fine for me however last week it suddenly stopped. When I run Yapeal it looks like it has run fine but nothing in the database updates. I have ran the check requirements and check privs scripts again and they still come back as ok.
Also nothing is in the log folders.
Is there anything I can do to get a clue as to what might have gone wrong?
Many thanks,
Nori. |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.11.25 18:15:00 -
[135] - Quote
As always make sure you are using the latest version and insure your config/yapeal.ini is up to date with any changes.
Try running it manually from command line might give you a few more clues also check that log files haven't filled up your disk space if you're using an older version before new logging system was added.
If all of that is good check that MySQL hasn't ran itself out of disk space with logs etc as I've seen some mis-configured ones if they use the defaults where they never clear out old logs etc when using ADDOdb instead of MyISAM like Yapeal does.
If after doing all of the above you still have problems try stopping any crontab/Scheduled Task of Yapeal and clearing out the API file and disk caches and the utilCachedUntil table then run Yapeal manually again and make sure the table refills right.
One last thing to check is if your API keys have expired.
If after all of the above it's still not work contact me via the project owner on Google Code at GMail. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Norian Lonark
Black Thorne Corporation Black Thorne Alliance
2
|
Posted - 2011.11.25 21:42:00 -
[136] - Quote
Hi Dragonaire,
Thanks for the tips how I got it working again was I deleted the API information from the utilRegisteredKey table, added it again (didnt change the API) and it started working again.
Will keep an eye on it over the next few days so may contact you for further help and many thanks for the kind offer.
Best regards,
Nori. |
Bado Roul
Deep Core Mining Inc. Caldari State
0
|
Posted - 2011.11.30 14:13:00 -
[137] - Quote
2-3 days ago my tables where finally up to date and the problem is still there.
Running yapeal.php 1h30min after last launch, I can see so many addRow in trace log (I stopped yapeal.php 3-4min after launching, normally takes 7min in this VPS):
Quote:-rw-r--r-- 1 bado bado 10151866 Nov 30 15:00 yapeal_trace.log -rw-r--r-- 1 bado bado 10485882 Nov 30 14:58 yapeal_trace.log.1
Quote:grep "YapealQueryBuilder::addRow" yapeal_trace.log* | wc -l 154118
Over 154000 "tries" |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.11.30 16:06:00 -
[138] - Quote
I would have been surprised if you didn't have lots of addRow() in trace consider it's called to store every single row (record) into the database that Yapeal does. Let me give an example with a very simple API like server/ServerStatus.
Once Yapeal decides the API needs to be updated it updates utilCachedUntil with new date-time with a short timeout in case something goes wrong on this update it can re-try in about 5 minutes. (BTW for this API that's it's normal cache time but on most APIs this is shorter than normal) It then tries to get the API data from the server. If it gets the data and you have database caching turned on it writes the XML to the utilXmlCache table and then does an upsert to the serverServerStatus table. Yapeal then figures out the new cachedUntil time it's going to use from XML (No it doesn't always use the one from XML directly because often the APIs lie about the true cache time and I got tired of all the errors) It then updates utilCachedUntil with the new date-time. So for just a single record API there are 3-4 writes to the database. 2-3 of those are overhead that don't change with the number of records that need to be updated.
Now let's think about an API like AssetList where there can be 1000s of rows of data for each of them addRow() is called. That's per char/corp as well so with just a dozen or so chars/corps you can easily end up with 10000 calls to it or more. The best way to get a good idea how many times it will be called look at how many rows (records) of data are in the all the API data tables plus the XML cache if you are using it and utilCachedUntil.
Now just to make something clear Yapeal doesn't actual store the data into the database on each call to addRow() it actually saves up the data and wraps up to 2000 of them by default into a single extended insert or upsert inside a database transaction to make them faster.
In all of the above I've ignored the selects that are needed on the database as well for Yapeal to do it's job. There are about six or so per API and do to the nature of them most can't be cached but for the ones that can Yapeal does some internal caching.
Let's now compare the number of writes Yapeal does to the database compared to say a blogging or forum site.
Let's say there are 10000 active people on the site and they are all quick readers and fast typists. They each take 5 seconds to read a post and 5 seconds to add their own and they are doing it constantly (Typical EVE General Discussion poster ). Lets guess each post causes three rows (records) in different tables to be written for each of them and it has to read from two tables to display one. So we have 10000 / 10 * 3 = 3000 rows being written each second and 10000 / 10 * 2 = 2000 reads. Now Yapeal can do that many writes on a single API for a single char/corp but a lot less reads. Most hosting site are setup at best to handle those type of loads and actually most of them are figuring on closer to 100 reads per write to the database since that's much more typical.
Given the above is it any wonder that when you start trying to add a few hundred or thousand users to Yapeal it's usually the database directly or indirectly via file I/O that can't keep up on most shared hosts or even a VPS? The above numbers are ignoring any overhead your applications might be adding which probably make up for some of the reads. The above numbers aren't going to be unique to Yapeal either but generally true for anything that tries to keep the all the API data in a database.
Anyway I hope that gives everyone a better idea of what you are truly asking of your poor server when working with the APIs Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Bado Roul
Deep Core Mining Inc. Caldari State
0
|
Posted - 2011.11.30 19:30:00 -
[139] - Quote
I'm here to find a solution, not to see if my VPS is good enough for Yapeal. Linode is one of the best VPS provider. I'm not familar with Yapeal code, so I don't know if so many addRow are good or not.
Anyway I've rebooted my machine to use Debian and this is the result for 1st run on a fresh install from Mercurial:
Quote:time ./yapeal.php
real8m2.422s user7m7.983s sys0m13.629s
Now there are 1000 rows in corpWalletJournal and another 1000 in corpWalletTransactions. Others are <10, not counting utilAccessMask and utilCachedUntil.
I had enabled logging in my.cnf, which I didn't notice to enable until now to debug. Next command look for one transactionID in my corpWalletTransaction:
Quote:root:/var/log/mysql# grep 2344954498 mysql.log |wc -l 2
There are two Queryinsert into `corpWalletTransactions` in the log and I see what you said about extend inserts.
Now I'll look for one refID in my corpWalletJournal:
Quote:root:/var/log/mysql# grep 5151289051 mysql.log |wc -l 1001
1001 extend inserts, which I think it is not desired behaviour.
If you have any suggestion or another test I can run, it will be welcome.
Thanks |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.01 05:32:00 -
[140] - Quote
First I don't want you to think I was saying there was anything wrong with your VPS I actual think Linode is a great service. I mostly did the post to give everyone some data to work with to understand the differences in the load that API applications in general and Yapeal in particular have so they have something to work with when trying to scale up with them and might have had some baring on your problems since I didn't know how many chars or corps you were trying to run it with.
You are right 1001 inserts on corpWalletJournal doesn't look right at all. I would say somehow the exit conditions aren't right and my fail safe is kicking in at the 1000-1001 mark and keeping it from becoming an infinite loop It's interesting to me that only the Journal was effected as transactions use the same code with only minor diffs but I'll look into it now that I've got a better idea what's happening so thanks for the extra logging it really helps. I'm also wondering if you are seeing the same thing happen with the same 2 APIs in the char section by chance? Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
|
Bado Roul
Deep Core Mining Inc. Caldari State
0
|
Posted - 2011.12.01 12:31:00 -
[141] - Quote
I only have one corp key because I was trying to isolate the problem. I'm on my machine again, I didn't update since 20h ago or so and, after that modification, yapeal.php took 35 seconds to finish. Now there are 2000 records on corpWalletTransactions and corpWalletJournal which seems fine
About API key in char section, I will add everything as I had before in my VPS and I will see.
Thanks! |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.01 16:15:00 -
[142] - Quote
Ok I'll push out new version with fix then and we'll see how it goes. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Bado Roul
Deep Core Mining Inc. Caldari State
0
|
Posted - 2011.12.02 09:42:00 -
[143] - Quote
With char wallets, CPU graph has spikes to 20-35%.
I checked last version and I see you've removed "FALSE" in char wallets. Testing now
Thanks! |
Somerset Mahm
Somer's Omnibus Exploration and Reclamation Cognitive Distortion
65
|
Posted - 2011.12.06 22:16:00 -
[144] - Quote
Hiya Dragonaire,
So I'm about to make the leap to new-API-key Yapeal. What's the preferred way to migrate data? Do I have to set up a new database and manually run queries to change formats? Will Yapeal upgrade schemas for me if I rerun createMySQLTables (and preserve data)?
Just taking it slow :P |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.06 23:49:00 -
[145] - Quote
There where some docs about converting around the time Yapeal changed over which you'll on the old thread in the archives and some stuff in the CHANGELOG as well but the basic process is to backup everything up and drop all the tables outside of char* and corp* ones then do upgrade and the missing tables will be recreated. Or you can just make a new database following the install instructions then copy over the data from char and corp tables from the old database. There are enough changes to the account tables to cause problems and they'll refill automatically anyway you don't want to keep them.
If you need any more help let me know. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Somerset Mahm
Somer's Omnibus Exploration and Reclamation Cognitive Distortion
65
|
Posted - 2011.12.07 04:25:00 -
[146] - Quote
Yeah, *walletJournal and *walletTransactions are the ones I'm really concerned with. Thanks! |
Louis Vitton
Kickass inc Controlled Chaos
2
|
Posted - 2011.12.07 04:50:00 -
[147] - Quote
Hey,
I am trying to install yapeal but when i run ".\checkForRequirements.php" in windows 7 all that happens is that it opens in notepad++ no messages are displayed. |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.07 04:58:00 -
[148] - Quote
That's because Windows doesn't think files ending in .php can be ran. Try it from command line and pass it to PHP by trying this:
php -f .\checkForRequirements.php
It should work right if you're PHP is setup correctly with that. Make sure you are following the instructions in INSTALL.md Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
Louis Vitton
Kickass inc Controlled Chaos
2
|
Posted - 2011.12.07 06:13:00 -
[149] - Quote
I dont have php installed locally but i did try to run it though xampp's portable one and still did the same thing. when i used php -f all it did was say didnt recognize it. Do i need to install php locally? |
Dragonaire
Corax. PURgE Alliance
21
|
Posted - 2011.12.07 06:24:00 -
[150] - Quote
Sounds like it's not finding you PHP either so yes you will need to have PHP installed to use it. Guess I thought it was pretty plain that a working local copy of PHP is a requirement for anything to work if you are going to try using it on your local computer
Edit : I'm really sorry if this comes across badly but what you are basically asking is "Do I really have to install the software needed to run your script?"
It was very hard not to answer that with something like this "No of course not you should instill Perl and Python and use them to write your own interpreter which converts it into Java code and run it that way"
Hopefully you understand I'm not really trying to be mean or anything but I just really never expect anyone to ask that question. Finds camping stations from the inside much easier. Designer of Yapeal for Eve API. Check out the Yapeal forum thread |
|
|
|
|
Pages: 1 2 3 4 [5] 6 7 8 9 10 11 12 13 14 15 16 17 18 .. 18 :: one page |
First page | Previous page | Next page | Last page |