Pages: [1] 2 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 3 post(s) |
Joshua Foiritain
Coreli Corporation Ineluctable.
923
|
Posted - 2014.08.08 11:25:00 -
[1] - Quote
TL;DR: I was wondering if anyone knew how the hash in the external killmail link is generated?
I've been trying to figure out a way to source kills my corp didnt get the final blow on that arent on zkb while retraining the accuracy the eve API/zkb API provide (item hierarchy, specialized hangars, proper item flags, etc). Obviously if members have the external killmail link they can simply post that to add the kill but this can be a lot of work depending on how much you kill during a fleet op. There also the problem that you cant pull these links from external killboards which makes them hard to track down afterwards. Copying killmails from other killboards aren't really an option as the mails lack a lot of info.
Pulling from killboards run by our allies is a problem because the quality of the killfeed varies, obviously any kill posted via mail will lack the detailed info i want, even if its API verified afterwards it doesnt update the kill with info from the API and then theres killboards whose killfeed data is malformed (rigs in cargo, weapons in wardrobe slots, etc).
So i've been looking at the external killmail link, if i pull the kill's api id + killmail from an external board i should technically have all the info needed to create a killmail link for the kill which i can then use to pull the kill directly from CCP, the only question is how the hash is generated; which algorithm is used, is it just the killmail or a killmail + salt?
I ran a killmail through a few of the standard algorithms but none of them matched the external killmail hash for that kill so either they're the wrong ones or extra info beyond the killmail is added prior to hashing. Does anyone know? DRUGS DRUGS DRUGS | Coreli Corporation, Serpentis Loyalists DRUGS DRUGS DRUGS | Small gang PVP & drug production DRUGS DRUGS DRUGS | Ship Replacement program DRUGS DRUGS DRUGS | Multiple roaming fleets per day |
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
3640
|
Posted - 2014.08.08 13:00:00 -
[2] - Quote
nope. It's not open, and I doubt it will become so. Because if it did, then you could scrape kills you have no 'right' to. Woo! CSM 9! http://fuzzwork.enterprises/ Twitter: @fuzzysteve on Twitter |
Joshua Foiritain
Coreli Corporation Ineluctable.
923
|
Posted - 2014.08.08 13:39:00 -
[3] - Quote
Steve Ronuken wrote:nope. It's not open, and I doubt it will become so. Because if it did, then you could scrape kills you have no 'right' to. Well you'd still need the kill's API ID so its not like being able to convert the mail into hash alone allows you to access any kill you want through the feed.
The only thing being able to hash the mails would allow you to do is verify the correctness of kills you already have the API ID for, if all the parties involved choose not to add their API keys to a killboard then there's still no way to scrape those mails.
Alternatively if the killlog api returned all kills an entity was involved on rather then just the ones the entity has the final blow on all of this wouldn't matter because the corp api would provide me with a 100% accurate corp killboard, but Im assuming there's technical limitations preventing that. DRUGS DRUGS DRUGS | Coreli Corporation, Serpentis Loyalists DRUGS DRUGS DRUGS | Small gang PVP & drug production DRUGS DRUGS DRUGS | Ship Replacement program DRUGS DRUGS DRUGS | Multiple roaming fleets per day |
Dragonaire
Here there be Dragons
56
|
Posted - 2014.08.08 22:39:00 -
[4] - Quote
I just will have to say this seems to get asked about every couple of weeks and the few times CCP has replied they say no they won't do it. You can do a search on the forums here to find the details on why.
I did want to say something on still allowing people to add kills by the kill mails they receive also. It came up the last time there was a bigger discussion about making all kill mails public. Approximately 99% of the kills get posted already so if you get tied in with a few well maintain kill boards you don't need to allow them. Understand in most cases there are at least 4 people that have access to the kills through the API. There's the victim, their corp, the person with final blow, and their corp. As long as one of those people is running an API key driven kill board it should show up through the intra-board feeds and there's no need to use cut and pasted mails any more which as you said don't have all the info you might want. By not letting people use the cut & paste mails you also solve your other problems. Just some things to think about. Finds camping stations from the inside much easier. Designer of Yapeal-á for the Eve API. Check out the Yapeal PHP API Library thread for more information. |
Joshua Foiritain
Coreli Corporation Ineluctable.
926
|
Posted - 2014.08.08 23:23:00 -
[5] - Quote
Dragonaire wrote:I just will have to say this seems to get asked about every couple of weeks and the few times CCP has replied they say no they won't do it. You can do a search on the forums here to find the details on why.
I did want to say something on still allowing people to add kills by the kill mails they receive also. It came up the last time there was a bigger discussion about making all kill mails public. Approximately 99% of the kills get posted already so if you get tied in with a few well maintain kill boards you don't need to allow them. Understand in most cases there are at least 4 people that have access to the kills through the API. There's the victim, their corp, the person with final blow, and their corp. As long as one of those people is running an API key driven kill board it should show up through the intra-board feeds and there's no need to use cut and pasted mails any more which as you said don't have all the info you might want. By not letting people use the cut & paste mails you also solve your other problems. Just some things to think about. I don't think you actually read my post before responding? It seems like you're responding to me wanting to be able to pull any kill i want, including ones that havent been made public by the people involved, which is obviously not what im looking for...
Also iirc the number is 95% and eve-kill/zkb no longer syndicates from other boards because of abuse. A problem my suggestion would also solve as they would now have a way to verify kills themselves without having to rely on the third party feeder board being trustworthy. DRUGS DRUGS DRUGS | Coreli Corporation, Serpentis Loyalists DRUGS DRUGS DRUGS | Small gang PVP & drug production DRUGS DRUGS DRUGS | Ship Replacement program DRUGS DRUGS DRUGS | Multiple roaming fleets per day |
Dragonaire
Here there be Dragons
56
|
Posted - 2014.08.09 03:29:00 -
[6] - Quote
Actually did read your post the point was CCP said NO. And one of the reason I've been told that some of the main boards stop accepting feeds is because some many of the boards were accepting copy and paste kill mails that were bogus and they got/get tired of dealing with them. Last I know though they still had outgoing feeds you can receive from so you might check if they still do that. Finds camping stations from the inside much easier. Designer of Yapeal-á for the Eve API. Check out the Yapeal PHP API Library thread for more information. |
Joshua Foiritain
Coreli Corporation Ineluctable.
926
|
Posted - 2014.08.09 08:51:00 -
[7] - Quote
Dragonaire wrote:Actually did read your post the point was CCP said NO. And one of the reason I've been told that some of the main boards stop accepting feeds is because some many of the boards were accepting copy and paste kill mails that were bogus and they got/get tired of dealing with them. Last I know though they still had outgoing feeds you can receive from so you might check if they still do that. I covered pulling from boards in my original post, hell my entire post is about the problems that come with that which can be resolved by being able to verifify the correctness of a kill via CCP's server, you bringing it up in your previous post is why i thought you didn't read mine
As for devs saying no, do you have any links to these threads? Eve's wonderful search function can only find 3 threads related to external kill mail links, one of which is mine, one is the announcement of their creation and the third is Squizz Caphinator suggesting killmail hashes in 2012 DRUGS DRUGS DRUGS | Coreli Corporation, Serpentis Loyalists DRUGS DRUGS DRUGS | Small gang PVP & drug production DRUGS DRUGS DRUGS | Ship Replacement program DRUGS DRUGS DRUGS | Multiple roaming fleets per day |
Dragonaire
Here there be Dragons
56
|
Posted - 2014.08.09 17:16:00 -
[8] - Quote
http://lmgtfy.com/?q=eve+online+public+kill+mails Second link under first result was one of them I was think of that happened just 3 months ago. There was some other threads before that I think in about the same time frame with some more in depth talk as well but that one is probably a fair summary of how it went. Finds camping stations from the inside much easier. Designer of Yapeal-á for the Eve API. Check out the Yapeal PHP API Library thread for more information. |
|
CCP FoxFour
C C P C C P Alliance
3391
|
Posted - 2014.08.09 20:27:00 -
[9] - Quote
Basically what has been said. No we will not tell you how to generate the hash.
zKillboard still syndicates their feeds from what I understand. I believe they have a publicly accessible stomp feed along with just their regular API. CCP FoxFour // Game Designer // @regnerba
|
|
Joshua Foiritain
Coreli Corporation Ineluctable.
929
|
Posted - 2014.08.10 21:49:00 -
[10] - Quote
Dragonaire wrote:http://lmgtfy.com/?q=eve+online+public+kill+mails Second link under first result was one of them I was think of that happened just 3 months ago. There was some other threads before that I think in about the same time frame with some more in depth talk as well but that one is probably a fair summary of how it went. I'm not asking for hidden killmails to be made public... I'm asking for a way to verify kills that provided to me from another killboards feed are real and accurate.
Dragonaire wrote:Edit: Other link you might find useful I know you said they had shutdown the feeds but I have a feeling it more likely something just changed and you'll need to /update your config http://eve-kill.net/?a=idfeedI tried a query manually in browser for a corp and it worked just fine so you really need to check your config as the feed is still active. If you use a different killboard you'll just need to do some searching around for how they do their feeds but I'll bet they are still available. They may have turned off receiving non API mails from outside but I'm sure they still make theirs available. I know there was talk about coming up with a better feed system a year or so ago but wasn't directly involved in them so don't know if they did anything or not. Eve-Kill/Zkillboard no longer pulls from third party boards, of course they still have a feed you can pull from, i mentioned this in my first post...
CCP FoxFour wrote:Basically what has been said. No we will not tell you how to generate the hash. :sadface: Any chance you could elaborate on why not though? I'm rather curious as to what the concerns behind releasing it are.
As Steve's scrapping concern isn't actually a risk, its more efficient to scrape every possible API/Hash combination then to try and generate every possible killmail combination, hash those, combine it with every possible API ID and scrape with that.
Dragonaire's concern that it could be used to find kills that arent public also isnt a risk because you need both the API ID and the killmail to access CREST, and having that information means the kill was made public by one of the people involved via their api.
CCP FoxFour wrote:zKillboard still syndicates their feeds from what I understand. I believe they have a publicly accessible stomp feed along with just their regular API. They still have a feed that can be pulled from but they no longer pull from other boards. Which means they end up missing kills which are available on the internet but not to them directly. The reason they stopped pulling kills from other boards is because of abuse, if they could use the CREST endpoint to verify that kills provided to them by other boards are real (since these boards provide the API ID and killmail, so you have all he data needed to access the CREST feed for confirmation except for the hashing method).
I recall you suggesting somewhere that killboards could provide the CREST link they receive from users on their kills like they do with killmails so that users could manually copy the CREST url into their board to transfer the kill over to their board. However killboards would only be able to provide the link if the kill was added via the link, if the kill was pulled directly from the API they could not provide the link because they dont have it. Wouldn't being able to generate the link if you already have the information required be logical next step?
Alternatively, any chance the text mails can be upgraded to include item flags so that we can see fleet hangars, specialized bays and such on them instead of everything being lumped under 'cargo'? DRUGS DRUGS DRUGS | Coreli Corporation, Serpentis Loyalists DRUGS DRUGS DRUGS | Small gang PVP & drug production DRUGS DRUGS DRUGS | Ship Replacement program DRUGS DRUGS DRUGS | Multiple roaming fleets per day |
|
Kali Izia
GoomWaffe Goonswarm Federation
1
|
Posted - 2014.08.10 23:43:00 -
[11] - Quote
It's definitely possible to generate the hash. I've just thrown up http://km.goomfleet.com/ as a proof of concept, but it's not terribly useful for anything as it is.
If CCP doesn't want it shared I'm not going to go against their will and post details about the hash, although I do agree that there isn't much harm in it. You do need both the kill ID and various information from the killmail to do anything so scraping kills etc shouldn't really be a concern. |
|
CCP FoxFour
C C P C C P Alliance
3391
|
Posted - 2014.08.11 10:40:00 -
[12] - Quote
Kali Izia wrote:It's definitely possible to generate the hash. I've just thrown up http://km.goomfleet.com/ as a proof of concept, but it's not terribly useful for anything as it is. If CCP doesn't want it shared I'm not going to go against their will and post details about the hash, although I do agree that there isn't much harm in it. You do need both the kill ID and various information from the killmail to do anything so scraping kills etc shouldn't really be a concern.
Hahaha guess it was only time until someone figured it out. If you feel like sharing feel free to. :) CCP FoxFour // Game Designer // @regnerba
|
|
Kali Izia
GoomWaffe Goonswarm Federation
6
|
Posted - 2014.08.11 11:57:00 -
[13] - Quote
Alright, so the hash is:
sha1(victimCharacterID + attackerCharacterID + shipTypeID + killTime)
Where:
- victimCharacterID = the character ID of the victim. If the characterID == 0, such as a kill for a POS mod or similar, this is the string "None" instead
- attackerCharacterID = the character ID of the attacker that got the final blow. If the characterID == 0, such as an NPC, this is the string "None" instead
- shipTypeID = the typeID of the victim's ship
- killTime = a 64-bit timestamp. CCP luckily doesn't store precision greater than 1 second otherwise we couldn't do this, so you can use the following math to convert from a regular unix timestamp: (unixtime * 10000000) + 116444736000000000
Just grabbing a random kill from the front page of zKill that doesn't have a CREST link, we'll test with kill 40583728: Our values are None + 93808108 + 24646 + 130521094800000000
sha1(None9380810824646130521094800000000) = efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d = http://public-crest.eveonline.com/killmails/40583728/efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d/ |
|
CCP FoxFour
C C P C C P Alliance
3394
|
Posted - 2014.08.11 13:51:00 -
[14] - Quote
Kali Izia wrote:Alright, so the hash is: sha1(victimCharacterID + attackerCharacterID + shipTypeID + killTime)Where:
- victimCharacterID = the character ID of the victim. If the characterID == 0, such as a kill for a POS mod or similar, this is the string "None" instead
- attackerCharacterID = the character ID of the attacker that got the final blow. If the characterID == 0, such as an NPC, this is the string "None" instead
- shipTypeID = the typeID of the victim's ship
- killTime = a 64-bit timestamp. CCP luckily doesn't store precision greater than 1 second otherwise we couldn't do this, so you can use the following math to convert from a regular unix timestamp: (unixtime * 10000000) + 116444736000000000
Just grabbing a random kill from the front page of zKill that doesn't have a CREST link, we'll test with kill 40583728: Our values are None + 93808108 + 24646 + 130521094800000000 sha1(None9380810824646130521094800000000) = efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d = http://public-crest.eveonline.com/killmails/40583728/efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d/
Think you are about to get a lot of love.
Mind posting it over here as well: https://wiki.eveonline.com/en/wiki/Equations CCP FoxFour // Game Designer // @regnerba
|
|
Joshua Foiritain
Coreli Corporation Ineluctable.
929
|
Posted - 2014.08.11 14:12:00 -
[15] - Quote
Kali Izia wrote:Alright, so the hash is: sha1(victimCharacterID + attackerCharacterID + shipTypeID + killTime)Where:
- victimCharacterID = the character ID of the victim. If the characterID == 0, such as a kill for a POS mod or similar, this is the string "None" instead
- attackerCharacterID = the character ID of the attacker that got the final blow. If the characterID == 0, such as an NPC, this is the string "None" instead
- shipTypeID = the typeID of the victim's ship
- killTime = a 64-bit timestamp. CCP luckily doesn't store precision greater than 1 second otherwise we couldn't do this, so you can use the following math to convert from a regular unix timestamp: (unixtime * 10000000) + 116444736000000000
Just grabbing a random kill from the front page of zKill that doesn't have a CREST link, we'll test with kill 40583728: Our values are None + 93808108 + 24646 + 130521094800000000 sha1(None9380810824646130521094800000000) = efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d = http://public-crest.eveonline.com/killmails/40583728/efd4bf9c4f2aee704d3f9a7f8ae0176a15eba19d/ Awww yissss, Pulling kills from EDK boards without having to worry about the quality of the data in the feed.
You are a hero. DRUGS DRUGS DRUGS | Coreli Corporation, Serpentis Loyalists DRUGS DRUGS DRUGS | Small gang PVP & drug production DRUGS DRUGS DRUGS | Ship Replacement program DRUGS DRUGS DRUGS | Multiple roaming fleets per day |
zKillboard
zKillboard.com
0
|
Posted - 2014.08.11 14:27:00 -
[16] - Quote
Kali Izia wrote:awesome killmail stuff/
I <3 you |
Masao Kurata
Z List
71
|
Posted - 2014.08.11 14:47:00 -
[17] - Quote
Now please add a random fraction of a second to kill any possibility of kill scraping. Or just outright salt it.
EDIT: I should be clearer about why this is, as it stands, a problem. 86400 seconds to a day is a fairly trivial number of requests if you're looking at one specific character, the main issue is that you have to combine two characters UNLESS it's an npc kill. Scraping NPC losses (CONCORD, sentry guns, pve accidents) is quite trivial and I'm sure someone could easily demonstrate that given the information in this thread. In the case of CONCORD and sentry guns that would give you a system and a narrow time range for a pvp killmail, which leaves the second character and ship type unknown. A bit more difficult to scrape but probably still possible especially combined with public data on active characters, of which there is some.
Combine these and you can scrape either pve losses or gank killmails, maybe some lowsec gate engagements too (although death by sentry there should be uncommon compared to highsec).
Salting the hash and including the salt in the kill information would do for verification. |
Kali Izia
GoomWaffe Goonswarm Federation
13
|
Posted - 2014.08.11 15:38:00 -
[18] - Quote
Masao Kurata wrote:EDIT: I should be clearer about why this is, as it stands, a problem. 86400 seconds to a day is a fairly trivial number of requests You also need the ID of the ship which adds to the number of requests. Since you mentioned NPC losses you could always reduce this to common ratting/missioning/whatever ships, but even then it's still a large number of requests.
Assuming you respect the rate limits, it would take about 2.5 hours per ship per character to scrape a single day for NPC losses. While it's possible to do in a day if you're targeting a specific character on a specific day and know what types of ships they might be flying, it's not something that can easily scale and the resulting intel is extremely minimal.
I can't see anyone putting in the time and effort just to figure out what fit you have on your ratting ship. |
Masao Kurata
Z List
71
|
Posted - 2014.08.11 18:31:00 -
[19] - Quote
The fact that it's possible to do at all should be concerning. Scraping kills is not intended to be possible. And I picked the example of a highsec ganker for a reason, there are only 5 or 6 truly good ganking ships. |
Squizz Caphinator
Happy Endings.
169
|
Posted - 2014.08.11 20:35:00 -
[20] - Quote
Masao Kurata wrote:The fact that it's possible to do at all should be concerning. Scraping kills is not intended to be possible. And I picked the example of a highsec ganker for a reason, there are only 5 or 6 truly good ganking ships.
Unless you already have the information as outlined above, you can't scrape killmails. If you do have that information, you already know everything but the killID - which is only of marginal value to those gathering intel. Various projects I enjoy putting my free time into: http://zkillboard.com | http://evewho.com | http://eve-kill.net | http://evechatter.com | http://skillq.net |
|
Dragonaire
Here there be Dragons
57
|
Posted - 2014.08.11 21:05:00 -
[21] - Quote
I know they are worried their 'super secret' fitting will get out because no one else could figure it out right Hate to break it to you but everybody already knows about it and it not that good Finds camping stations from the inside much easier. Designer of Yapeal-á for the Eve API. Check out the Yapeal PHP API Library thread for more information. |
Masao Kurata
Z List
71
|
Posted - 2014.08.12 20:25:00 -
[22] - Quote
Squizz Caphinator wrote:Masao Kurata wrote:The fact that it's possible to do at all should be concerning. Scraping kills is not intended to be possible. And I picked the example of a highsec ganker for a reason, there are only 5 or 6 truly good ganking ships. Unless you already have the information as outlined above, you can't scrape killmails. If you do have that information, you already know everything but the killID - which is only of marginal value to those gathering intel.
Yes you can, you didn't read my post.
And of course it's not really all that harmful but this API is not meant to be possible to scrape, yet it is under certain circumstances. |
Squizz Caphinator
Happy Endings
169
|
Posted - 2014.08.12 21:43:00 -
[23] - Quote
Masao Kurata wrote:
Yes you can, you didn't read my post.
And of course it's not really all that harmful but this API is not meant to be possible to scrape, yet it is under certain circumstances.
http://www.amazon.com/Aluminum-Foil-Deflector-Beanie-Protection/dp/1581603762 Various projects I enjoy putting my free time into: http://zkillboard.com | http://evewho.com | http://eve-kill.net | http://evechatter.com | http://skillq.net |
Mazaron
DeathByDestruction
0
|
Posted - 2014.08.13 07:46:00 -
[24] - Quote
Didn't CCP say that since zkillboard has like 99.7% of all killmails they were going to make them public anyway? |
Joshua Foiritain
Coreli Corporation Ineluctable.
930
|
Posted - 2014.08.13 08:30:00 -
[25] - Quote
Mazaron wrote:Didn't CCP say that since zkillboard has like 99.7% of all killmails they were going to make them public anyway? No that plan was cancelled because the owners of the remaining mails did not want their failures to become public DRUGS DRUGS DRUGS | Coreli Corporation, Serpentis Loyalists DRUGS DRUGS DRUGS | Small gang PVP & drug production DRUGS DRUGS DRUGS | Ship Replacement program DRUGS DRUGS DRUGS | Multiple roaming fleets per day |
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
3657
|
Posted - 2014.08.13 12:05:00 -
[26] - Quote
95%.
While I personally support making them all public, I can understand the reasons for not doing so. And I wouldn't want them immediately public.
There are people who don't put their kills on public boards, because it leaks useful information. Like when they're active.
Sure, the loss mail might show up, but there are some targets who don't have killmails set up on a public killboard. Woo! CSM 9! http://fuzzwork.enterprises/ Twitter: @fuzzysteve on Twitter |
Masao Kurata
Z List
71
|
Posted - 2014.08.13 16:09:00 -
[27] - Quote
Okay, here's an anecdotal statistic for you: the alt I pvp on most has to go back 21 kills to find one that's on zkillboard. |
Aquila Sagitta
Blue-Fire
422
|
Posted - 2014.08.19 18:28:00 -
[28] - Quote
And how many kills total does your alt have? 21 kills isn't much in the grande scheme of things thats like a week or less of pvp Blue-Fire Best Fire |
Masao Kurata
Z List
72
|
Posted - 2014.08.19 21:19:00 -
[29] - Quote
The point is that none of those are on any killboard. This is fairly typical, I don't have an actual statistic for how many of his victims have registered APIs on killboards because of complications like kills I intentionally posted, fleet kills and such but it's very low. Potentially exposing kills by scraping is not a non-issue. |
Joshua Foiritain
Coreli Corporation Ineluctable.
931
|
Posted - 2014.08.20 14:14:00 -
[30] - Quote
Masao Kurata wrote:The point is that none of those are on any killboard. This is fairly typical, I don't have an actual statistic for how many of his victims have registered APIs on killboards because of complications like kills I intentionally posted, fleet kills and such but it's very low. Potentially exposing kills by scraping is not a non-issue. Scraping kills was already possible before. Theres a limited number of API Id's and a limited number of possible Hashes. DRUGS DRUGS DRUGS | Coreli Corporation, Serpentis Loyalists DRUGS DRUGS DRUGS | Small gang PVP & drug production DRUGS DRUGS DRUGS | Ship Replacement program DRUGS DRUGS DRUGS | Multiple roaming fleets per day |
|
|
|
|
Pages: [1] 2 :: one page |
First page | Previous page | Next page | Last page |