| Pages: 1 2 [3] 4 5 6 7 8 9 10 11 .. 11 :: one page |
| Author |
Thread Statistics | Show CCP posts - 0 post(s) |
Dragonaire
Here there be Dragons
56
   |
Posted - 2014.08.08 15:48:00 -
[61] - Quote
Tiberius Zol - Sorry you lost all the data without backups but there is a reason I called it init. I also clearly state in the instructions to make backups before starting. Something I will try to do going forward which I really have been mean to do is make an update SQL with just the changes. I'll have to come up with a naming system that make sense and is easy to understand as well but it is something that will be needed more and more as time goes on. I just had another idea how to do it as well that might be easier to maintain so give me a couple days and I'll see if it might work the way I think it could.
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Tiberius Zol
27
|
Posted - 2014.08.08 16:49:00 -
[62] - Quote
nevermind. had a backup from yesterday. was the first shock only. :) i will keep in mind to change all sql creation files until we have a update mechanism. :) |
Legedric Striker
Blue Republic
RvB - BLUE Republic
33
|
Posted - 2014.08.08 19:31:00 -
[63] - Quote
Well, of course there is always the possibility to just take the SQL files from the most recent Yapeal version, take your favourite version control (like SVN or git) and see for the diffs yourself and just adopt the changes from there. No need to recreate the tables at all ;)
Of course you should know what you are doing 
Join R-v-B-- The MOST active PVP community in EVE!
EVE-Skillplan.net - Plan your training for EVE Online |
Tiberius Zol
27
|
Posted - 2014.08.08 19:53:00 -
[64] - Quote
yeah.. learned about it.. never do things after 30 hours without sleep. ;-) |
Legedric Striker
Blue Republic
RvB - BLUE Republic
33
|
Posted - 2014.08.08 20:39:00 -
[65] - Quote
Tiberius Zol wrote:yeah.. learned about it.. never do things after 30 hours without sleep. ;-)
Seems like you are doing that pretty often as I think I may have read that you were saying this a couple of days ago
Join R-v-B-- The MOST active PVP community in EVE!
EVE-Skillplan.net - Plan your training for EVE Online |
Jack Haydn
Valar Morghulis.
Goonswarm Federation
52
|
Posted - 2014.08.09 20:57:00 -
[66] - Quote
Dragonaire wrote:Actual Yapeal does locking on a per API basis which allows multiple instances to run without stepping on each others toes just like it always has for a long time. AbstractCommonEveApi::gotApiLock() handles getting the lock. Using the database for locking is faster plus more cross platform compatible for what it needs. It also using transactions for all deletes and upsert so it makes sure multiple table APIs are updated correctly and keeps everything ACID. So you shouldn't need to do any extra locking in your crontab. You said it would cause weird behavior, have you actually experienced this or just assumed because you didn't see any locking that it would?
I'm a little puzzled by what you mean when you say 'delete APIs' maybe you can explain what you mean just a little more.
Sorry for the long delay, was quite busy.
I have not looked into what kind of locking yapeal does specifically, so I did not notice any row or table level locking. Somer did recommend using a file system based lock file though, since that is what he does aswell. I know little about yapeal, but I guess Somer kind of knows what's going on there. So I'm the wrong one to ask what exactly is happening. Maybe the locking logic is bugged, maybe Somer and I have set something up utterly wrong.
What I do know, is that CCP eventually blocked me and I was getting API data in stretches with lots of holes inbetween. Once I manually went through the database, weeded out a lot of expired/changed/deleted API keys and set my cron interval to 15 minutes, everything is running smooth again and I'm getting full API coverage. This kind of self-maintenance would be nice to have in the API library. Maybe it's even there, but then apparently bugged or I'm dumb.
This brings me to my second suggestion, about deleting APIs: AFAIK yapeal only provides methods to make API keys inactive (setting the isActive flag to zero), but no methods to actually completely remove API keys from the system. |
Dragonaire
Here there be Dragons
56
|
Posted - 2014.08.09 22:15:00 -
[67] - Quote
Jack Haydn - Sound like your application needs to do a better job of check keys before they get added to utilRegisteredKeys but to why Yapeal doesn't delete 'bad' keys and just deactivates them is randomly the API servers will return error codes suggesting it's a bad key. If it actual did delete them every time the API reported a problem you'd very soon end up with no keys at all.
My suggestion is you have a crontab that check for deactivated keys and use oneShot() calls to APIKeyInfo to check the keys are still valid etc and decided what you want to do directly. Another reason Yapeal doesn't do it automatically is some sites might want to send eve-mail or do something else to let the site admin or user know there is a problem. By simple making it not active the problem can be corrected and reactivate it. I try NOT to decide how things like that have to be handled because that's not Yapeal job that's the application's job Yapeal just tries to keep things from happening like your IP from being banned. Now if you find something in the logs where it's not deactivating a key when it should be do let me know and I'll make sure it gets fixed.
What might be helpful to figure out why you ended up getting banned, which by the way is very hard to do, except if there's something true wrong with your app setup if we could see the log files. There should be no way Yapeal would be breaking any of the API caching rules which would cause this but if somehow it is I would have to see the log files to figure out where the error might be. I would suggest instead of trying to paste all or part of your log file here and fight the forum formatting etc that you put it on Pastebin or something like it and just include a link. If you prefer you can also E-mail it to me at the E-mail listed on GitHub just make sure you ID yourself so it doesn't end up in my junk bin .
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Jack Haydn
Valar Morghulis.
Goonswarm Federation
52
|
Posted - 2014.08.09 23:00:00 -
[68] - Quote
Okay, fair enough, just deactivating them in the automatic process is reasonable. But apparently that failed somehow. Unfortunately, the bad keys are now purged and the logs were not set to debug or similar, so at this point there is not much I can provide for debugging purposes.
What I am asking for regarding the deletion of keys is a method which facilitates deletion. Not automatically triggered, but just a function I can call, pass it a keyid, and it'll remove the key and all relevant entries. It's quite a pain to figure out all the util and account joins needed to remove all the entries associated with it and hopefully not miss one ;)
IMO the library should abstract maintenance of the stuff as much as possible, so that the developer does not need to look into all the details. Otherwise one might just write the thing on their own :)
Another possible idea: In the spirit of above paragraph, a suitable abstraction for api key maintenance would be to provide a function which takes an API keyID and returns the state of the API. It can be good or display the number of times or the time interval it consecutively failed to receive a valid response from the API server. Then the developer can easily call that, and send out email warnings etc. Figuring out the inner workings of yapeal key maintenance is not needed anymore, then, in combination with a library-provided delete method. |
Dragonaire
Here there be Dragons
56
|
Posted - 2014.08.10 05:41:00 -
[69] - Quote
delete from utilRegisteredKey where isActive = 0;
Now if you want to remove all the data from the API tables that is a little more complex and I can give you some pointers on that but if you are already selecting the data to show it in your application you know how to do it really. Just change your selects into deletes and you're done. All the joins etc will be the same or nearly the same and I'm sure you can figure it out.
Let me quote something from the main page of Yapeal here.
Quote:Yet Another Php Eve Api Library also know as Yapeal which can be used to access the Eve Online API data and place it into a database.
I think it does that and I'll assume you did too or you wouldn't have used it. The idea of Yapeal is not to abstract everything database away for you but to simple have a auto-magically updating database so you don't have to deal with all the details of the XML. The original idea of Yapeal is the database is the interface that you should only need to deal with it. That has changed in some small ways but in the end I still use that as the guide to what is done and included in Yapeal. It is NOT Yapeal's job to handle everything related to the database because it really can't. Every application or even programmers writing the same application are going to do things differently and what might work great for one type of application or one programmer could bring tears to the eyes of another programmer. The code in some of the older versions of Yapeal and some of my other projects have been know to nearly do this at times actually for me when I look at them now
Think about what Yapeal does do and really how simple it is to work with.
- You have one small config file that is set and forget.
- You turn on and off which APIs it does anything with from a single column in one table (utilEveApi) isActive.
- You only have to add site user API key information to one table (utilRegistereedKeys) and Yapeal takes care of the details afterwards.
- If you need to stop Yapeal from getting some site user data once again there's only one table you need to work with utilRegisteredKey.
- Do a one time setup of a crontab or scheduled task on a preferred one minute timer.
- Yapeal goes to work making sure all the API data you have requested is as up to date as is allowed by the servers.
So in the end you only have two tables in the database your application has to manage so Yapeal can update over 100 tables for all your site users. I think that a pretty good deal.
Now with that said if you look I did start some work on a class to help manage the utilRegisteredKey table which you'll find in bin/ directory. It currently just helps you add a new key but I do plan on it being a more complete management tool for people to use on that table. If you want to add to it and share it with everybody please do as I'm not sure when I might get back to working on it as I have many other things to do in Yapeal and otherwise. I also would like to see something to help manage the utilEveApi table to make turning APIs on and off easier with a more OOP interface etc.
I do want to thank you for the feedback and ideas even if I probably wouldn't change Yapeal in the ways you want and as always thank you for using Yapeal.
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Dragonaire
Here there be Dragons
56
|
Posted - 2014.08.11 20:09:00 -
[70] - Quote
Just a quick note I've added corp/Facilities for those of you working with IndustryJobs. Note that I include an update SQL file in bin/sql/update/. It's mostly an early idea about doing this so will have some changes to it in the future but at least it can be used manually for now with edits until I develop the code to use it more directly.
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Kolb
Novaku Inc
Novaku Alliance
3
|
Posted - 2014.08.12 15:30:00 -
[71] - Quote
I wrote a slightly fancy bash script for my yapeal cron job:
http://pastebin.com/f9KJUBhd
It keeps the cron's logfile around 1 MB is size by trimming out the oldest records.
| It checks if the API server is working if so,
|---- it checks if Yapeal is running if so,
|-------- it stops the script from starting Yapeal again.
|---- if Yapeal isn't running it processes any number of scripts you wish to execute in series with yapeal somewhere in the series.
| if the API server isn't responsive it stops the script
Happy yapealing,
Kolb
|
Dragonaire
Here there be Dragons
57
|
Posted - 2014.08.12 17:43:00 -
[72] - Quote
Kolb - Thanks for sharing just have a few suggestions.
- First instead of trying to manage the log files yourself look at logrotate it's made to handle that type of stuff and is what most people use.
- Instead of copying rows from one table to another you could look at replacing utilRegisteredKey with a view or change you app so it uses the keyID as a foreign key in your table. You might find CreateCustomTables.sql helpful here.
- Why check for the API server when Yapeal already does that? It's a waste of your time writing code to try replacing something Yapeal already handles.
- Once again THERE IS NO PROBLEM WITH MORE THEN ONE YAPEAL RUNNING AT THE SAME TIME IT WAS MADE TO DO SO. It does locking on a per API and keyID basis and no additional locking is needed or management is needed. If you want to have Yapeal NOT be able to keep up with increasing load continue NOT letting this happen and it'll get slower and slower with more users. Running Yapeal with the one minute crontab is basically a simple way to make it act like it's threaded which can't be done in a cross platform way at this time in PHP or I'd use it. Yapeal spends 99% of it's time waiting on either your network connection to get the XML or your database/ filesystem to store it.
In summary stop re-inventing the wheel.
Stop try to second guessing over 5 years of Eve API development experience that has gone into Yapeal from me and others. Everything you are trying to do I've tried in Yapeal at some point in the past and if it's NOT there now there is a reason for it. The reason is usually I found a better way or it didn't work or even that it simple wasn't needed.
Just a suggestion try it my way. One of three things will happen.
- You'll find I'm right and it does work better then what you've been trying.
- You won't be able to measure any difference and my way is simpler.
- You find something in Yapeal that can be improved because you have tested it both ways and can show that and everyone benefits when I give you credit when adding it to Yapeal.
One final thing don't really take the above personally in anyway I've just been down this road a few times and needed to blow off some steam about it. I'm glad you want to improve things and help others and if you can show me I'm wrong on something that great as well.
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Kolb
Novaku Inc
Novaku Alliance
3
|
Posted - 2014.08.12 18:57:00 -
[73] - Quote
Dragonaire wrote:Kolb - Thanks for sharing just have a few suggestions.
- First instead of trying to manage the log files yourself look at logrotate it's made to handle that type of stuff and is what most people use.
- Instead of copying rows from one table to another you could look at replacing utilRegisteredKey with a view or change you app so it uses the keyID as a foreign key in your table. You might find CreateCustomTables.sql helpful here.
- Why check for the API server when Yapeal already does that? It's a waste of your time writing code to try replacing something Yapeal already handles.
- Once again THERE IS NO PROBLEM WITH MORE THEN ONE YAPEAL RUNNING AT THE SAME TIME IT WAS MADE TO DO SO. It does locking on a per API and keyID basis and no additional locking is needed or management is needed. If you want to have Yapeal NOT be able to keep up with increasing load continue NOT letting this happen and it'll get slower and slower with more users. Running Yapeal with the one minute crontab is basically a simple way to make it act like it's threaded which can't be done in a cross platform way at this time in PHP or I'd use it. Yapeal spends 99% of it's time waiting on either your network connection to get the XML or your database/ filesystem to store it.
In summary stop re-inventing the wheel. Stop try to second guessing over 5 years of Eve API development experience that has gone into Yapeal from me and others. Everything you are trying to do I've tried in Yapeal at some point in the past and if it's NOT there now there is a reason for it. The reason is usually I found a better way or it didn't work or even that it simple wasn't needed. Just a suggestion try it my way. One of three things will happen.
- You'll find I'm right and it does work better then what you've been trying.
- You won't be able to measure any difference and my way is simpler.
- You find something in Yapeal that can be improved because you have tested it both ways and can show that and everyone benefits when I give you credit when adding it to Yapeal.
One final thing don't really take the above personally in anyway I've just been down this road a few times and needed to blow off some steam about it. I'm glad you want to improve things and help others and if you can show me I'm wrong on something that great as well.
I'm not second guessing you Dragonaire, I wrote the bash for my specific needs in my environment is all. I put some effort into my cron and thought it might be something others could use in their work or even as an educational example.
:-)
I have a reason for not wanting to run my three scripts over themselves and it doesn't involve me worrying about the state of the yapeal tables but wanting yapeal to be finished with it's work before I continue on with the following steps in my process. Again this script isn't just about kicking off yapeal, I wanted to keep all of my tools running during an API outage.
I wasn't judging or commenting about the state or capabilities of yapeal, just building a tool for my needs and sharing my lessons learned with others out there that might not have experience with building a bash script to run as a cron. |
Tiberius Zol
27
|
Posted - 2014.08.13 19:09:00 -
[74] - Quote
Is the Posfuel stuff broken? I have two old entries in corpPosfuel but no actual data. |
Dragonaire
Here there be Dragons
57
|
Posted - 2014.08.13 23:27:00 -
[75] - Quote
Make sure they aren't for the same POS the cache files are on a per KeyID/characterID basis. So if you have say 2 CEO keys or a CEO and Director key for the same corp you'll have 2 cache files but only one set of data in the DB.
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Tiberius Zol
27
|
Posted - 2014.08.14 06:10:00 -
[76] - Quote
I have 4 Characterkeys and 2 Corpkeys (1 Director, 1 CEO but different Corps and different Chars)..
I also have no Membertracking for the Corp with Director Key.. on the old yapeal both works. |
Dragonaire
Here there be Dragons
57
|
Posted - 2014.08.14 07:02:00 -
[77] - Quote
Tiberius Zol - Do all the POSes show in StarbaseList table?
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Tiberius Zol
27
|
Posted - 2014.08.14 07:44:00 -
[78] - Quote
yes. |
Dragonaire
Here there be Dragons
57
|
Posted - 2014.08.14 19:00:00 -
[79] - Quote
Tiberius Zol - So it seems to be working here fine but I only have a single POS to work with. I'll go through some checks you can do to maybe narrow down where something might be breaking.
- If you look in cache/corp you should find a couple of StarbaseList*.xml files which list all the POSes you see in the DB table.
- You should see one StarbaseDetail*.xml for each POS as well.
- Looking at the StarbaseDetail*.xml files they should look something like http://pastebin.com/hVXtxk5Y. Notice if there is a 'fuel' element and make sure there are rows in it.
If all the above is good then you will next need to check log/yapeal.log and look for the messages on the POS stuff. Easy way to find them is do a find/search for 'Corp/Starbase'. Should see where StarbaseDetail calls StarbaseList to make sure it's up to date then processes each POS in turn. Make sure it doesn't report invalid XML when checking with XSD. Next it inserts the info into the 'corpStarbaseDetail', 'corpFuel', 'corpCombatSettings', and 'corpGeneralSettings' tables. Look for any exceptions especially having to do with the inserts. Also note how many rows it says for each table with fuel being the only one that should have more than a single row.
If you do find something in the log if you can put is up on Pastebin or some where I can take a look I'll try figuring out what's causing the issue. If you don't find anything when you look at the log you might still put it up somewhere where I can have a look at it as I might spot something that looks wrong to me still.
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Tiberius Zol
27
|
Posted - 2014.08.15 06:33:00 -
[80] - Quote
Checked several cache files now. I have no cachefile for my online pos. there several cache files for my offline poses including one offline pos with some fuel left (this entrys are in my db, so its fine).
I have checked the logfile also. I found only one entry with an insert into my db.. and this entry is from 2014-08-09.
It wrotes only the two lines from the offline pos with the fuel left.
here a pastebin from the logpart: http://pastebin.com/SNmeztzA |
Tiberius Zol
27
|
Posted - 2014.08.15 15:24:00 -
[81] - Quote
Found another Problem with WalletTransactions:
[2014-08-15 16:52:50] yapeal.WARNING: Failed to upsert data from Eve API char/WalletTransactions for 93116526 on account 1000 {"exception":"[object] (PDOException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'typeName' at row 72 at /htdocs/eve/yapealneu/lib/Database/EveApiToolsTrait.php:107)"} []
typeName in the old yapeal: varchar(255)
typeName in the new yapeal: char(50) |
Dragonaire
Here there be Dragons
57
|
Posted - 2014.08.15 20:35:00 -
[82] - Quote
Based on my post from yesterday I decided to start putting together a little guide to help people when trying to troubleshoot Yapeal. I don't know how useful it might be for people so ideas and feedback on how to improve it are welcome. Also like to hear ideas for how to continue on or does it need to continue? Anyway here the link:
https://github.com/Dragonrun1/yapeal/wiki/Troubleshooting-Tips
Tiberius Zol - Thanks for report on WalletTransactions problem I started issue for it.
https://github.com/Dragonrun1/yapeal/issues/33
I'm still looking at the other thing with Starbase but let's try one thing that might help on getting a little more complete log message for all you POSes. It looks as though corpStarbaseList for ownerID = 98067338 wasn't expired at the time you took that log so let's 'force' all the Starbase stuff to need refreshing at the same time. This may work better for you if you temporarily stop any crontabs etc and run Yapeal manually.
- Go into the utilCachedUntil table and delete all the StarbaseList and StarbaseDetails rows.
- You can also optional delete the cache files for them in cache/corp/ but Yapeal should use or replace them as needed.
- You might also find truncating the log/yapeal.log file useful.
- Now run Yapeal manually on CLI with something like 'php bin\yapeal.php'.
Once it finishes take a look once again at the tables that didn't seem to be updating and see if they changed also look at your log and see if it now shows anything different like trying to add the other POS that is missing. If you can Pastebin the section from the new log so I can compare it with to old one that would be great as well.
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Tiberius Zol
27
|
Posted - 2014.08.15 22:14:00 -
[83] - Quote
Ok what i made now:
- renamed log/yapeal.log (for creating an empty one)
- deleted all Starbase stuff from utilCachedUntil
- deleted all Starbase stuff in cache/corp/
- run Yapeal manually
after it.. no change (one error in the logfile cause yapeal didn't find the deleted xml)
if have paste the starbase stuff from the new yapeal.log
i also runned https://api.eveonline.com/corp/StarbaseDetail.xml.aspx with the same keyID/vCode for the active pos and pasted it
and i pasted my data from starbaselist
all in one pastebin here: http://pastebin.com/Dp0KbZ6t
as a sidenote.. there is only one starbaseDetail xml file in cache/corp (with the offline pos with little fuel left) |
Dragonaire
Here there be Dragons
57
|
Posted - 2014.08.19 20:23:00 -
[84] - Quote
Tiberius Zol - I've been looking at stuff for last couple days and just not seeing anything but I just pushed an update that will add into the XML via processing instruction a json copy of the arguments used when retrieving it from the API server which might help one of us figure out why you're have problems. I did setup a new test POS myself and everything seems to be working fine with the APIs. Try the update and clear the cachedUntil table and the FS cache and see if the XML with the new info makes a difference because without something more to work with not sure what I can do to help you 
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Tiberius Zol
28
|
Posted - 2014.08.19 20:48:00 -
[85] - Quote
did you try more than one pos? maybe there is a problem that its not loop over all poses? but i will try the update now. |
Tiberius Zol
28
|
Posted - 2014.08.19 21:01:00 -
[86] - Quote
OK..deleted all entries in StarbaseList, StarbaseDetail and Fuel, deleted starbase entries in cachedUntil and Cache also.
runned yapeal manually..
- all entrys in starbaseList seem to be ok
- theres only one entry in starbaseDetail (should be one for each pos i think?)
- posFuel have only entry for this entry...
as a sidenote.. this pos entry is for the first entry in starbaseList. seems really a loop problem?! |
Dragonaire
Here there be Dragons
58
|
Posted - 2014.08.19 23:46:00 -
[87] - Quote
Tiberius Zol - You are correct it is a loop problem more that it's missing a loop. Person that worked on API hadn't done any of the more complex ones before and I missed it during QA also. We only had single POS to test with so it seemed to work. Now that I know where the problem is should have it fixed in a day or two at most.
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Dragonaire
Here there be Dragons
58
|
Posted - 2014.08.20 01:40:00 -
[88] - Quote
Tiberius Zol - Give the update to DatabaseDetails a try and see if it fixes your issue.
https://github.com/Dragonrun1/yapeal/issues/35
Finds camping stations from the inside much easier.
Designer of Yapeal-á for the Eve API.
Check out the Yapeal PHP API Library thread for more information. |
Tiberius Zol
28
|
Posted - 2014.08.20 07:11:00 -
[89] - Quote
I'm in the office now. Will give it a try later this day. |
SCORPl0
Foundation Cutting-Edge
Mordus Angels
1
|
Posted - 2014.08.20 12:39:00 -
[90] - Quote
Hello,
following steps to install and start to use yapeal but it looks like the wiki is obsolete and concern old yapeal version.
I am in the step where testing yapeal ... but ... no yeapeal.php in install base dir ...
So now what to test and star please ?
ps: installed from git repo |
| |
|
| Pages: 1 2 [3] 4 5 6 7 8 9 10 11 .. 11 :: one page |
| First page | Previous page | Next page | Last page |