Pages: [1] :: one page |
|
Author |
Thread Statistics | Show CCP posts - 0 post(s) |
SJ Astralana
Syncore
142
|
Posted - 2016.09.24 04:37:21 -
[1] - Quote
Re this endpoint: https://crest-tq.eveonline.com/market/10000043/orders/all/?page=1
I have some uncontroversial code shown below, but where 50,000 records in a 5mb payload will load in a browser in under ten seconds, it takes over two minutes. Anyone run into this?
public static string Load(string url) { var req = (HttpWebRequest)WebRequest.Create(url); var response = req.GetResponse(); using (var stream = response.GetResponseStream()) { if (stream == null) return null; using (var buffer = new BufferedStream(stream)) { var reader = new StreamReader(buffer); return reader.ReadToEnd(); } } }
Hyperdrive your production business: Eve Production Manager
|
Steve Ronuken
Fuzzwork Enterprises Vote Steve Ronuken for CSM
6192
|
Posted - 2016.09.24 14:42:25 -
[2] - Quote
I've been using python to parse them, and I'm not seeing any real issue. Downloading, parsing, and dumping them _all_ into a file, in 30-120 seconds.
Woo! CSM XI!
Fuzzwork Enterprises
Twitter: @fuzzysteve on Twitter
|
Zifrian
Distortion. Amplified.
1779
|
Posted - 2016.09.24 22:34:57 -
[3] - Quote
You might want to check different options to the settings for HttpWebRequest. I noticed it was really slow until I figured maybe it was something with .Net instead of CCP.
Here are some settings that helped me with my CREST calls (in vb.net):
ServicePointManager.DefaultConnectionLimit = 20 ServicePointManager.UseNagleAlgorithm = False ServicePointManager.Expect100Continue = False
Dim request As HttpWebRequest request = DirectCast(WebRequest.Create(myUri), HttpWebRequest)
request.Method = "GET" request.Proxy = nothing request.PreAuthenticate = True request.UnsafeAuthenticatedConnectionSharing = True
Hope it helps.
GÇ£Any fool can criticize, condemn, and complain - and most fools do.GÇ¥ - Dale Carnegie
Industry guy, third-party developer, jack-of-all-trades - master of none
Maximze your Industry Potential! - Download EVE Isk per Hour!
|
SJ Astralana
Syncore
142
|
Posted - 2016.09.25 01:49:01 -
[4] - Quote
Zifrian wrote: You might want to check different options to the settings for HttpWebRequest.
Strange, no change...
Hyperdrive your production business: Eve Production Manager
|
SJ Astralana
Syncore
142
|
Posted - 2016.09.25 22:04:13 -
[5] - Quote
Bullseye: req.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
I looked at the traffic in wireshark and it looked identical, which pointed me to the browser must be receiving less data over the wire.
Hyperdrive your production business: Eve Production Manager
|
|
|
|
Pages: [1] :: one page |
First page | Previous page | Next page | Last page |