EVE Search - Replacing the physical clusters virtually...

All Channels

EVE General Discussion

Replacing the physical clusters virtually...

» Click here to find additional results for this topic using Google

Monitor this thread via RSS [?]

Pages: [1] :: one page
Author	Thread Statistics \| Show CCP posts - 0 post(s)
Popsikle Minmatar Re-Awakened Technologies Inc Electus Matari	Posted - 2007.09.24 00:41:00 - [1] A portion of my job is running VPS's, VM's, and load balancing for Itra/Internet applications and such. An Opensource technology we are using (well researching) is Xen. Xen gives us the ability to put X machines in a cluster and run Y operating systems off of it. One of the benefits of XEN over OpenVZ or any of the other VM applications is the ability to split a Virtual Machine over multiple Physical CPU's. I know right now you guys say with your architecture its not possible to split a node over multiple CPU's, but XEN would be able to change that. Its pretty neat little software and It might be something you guys should look in to ;) As an example if we start with a test bed of 16 Physical Computers, all interconnected via GigE we can end up with a whole LB'd Web cluster. We can set those 16 servers to act as: 1 Virtual MySQL server (split over 4 physical machines) 4 Virtual Web servers (split over 8 physical machines) 2 Virtual Load balancers (split over 4 physical machines). Or any combination of the above. We can add/remove physical servers to the pool of any virtual server on the fly to allow for spikes in processing needs or for failover/backup reasons. Im still only playing around with it, so my exp with it could still be pretty far off its capabilities but it seems like a pretty cool little technology that I figured you guys (or even anyone who plays eve, as alot of us are nerds at <3) would want to play around with ;) PS. If anyone has any exp running/playing with XEN at all, I would like to hear about it, so I can figure out if it will meet my needs for anything other then just playtime ;) ____ <t20> i want to be in a manager potition at Hooters <SaraDawn> Garthagk, do you have it up ? <Garthagk> I can get it up anytime.
Arana Tellen Gallente The Blackguard Wolves Knights Of the Southerncross	Posted - 2007.09.24 00:52:00 - [2] They are moving over to this type of thing next year. --------------------------------- Core 2 Duo E4300 1.8ghz @ 3ghz, 2GB Gskill DDR2 5400 @ 800mhh 4-4-4-12, Abit fatality mATX F-I90HD @ 334mhz, 8800GTS 320mb 2x250GB 7200.10s Raid 0, Vista 64 Home.
An Anarchyyt Gallente Sublime.	Posted - 2007.09.24 01:01:00 - [3] I believe this is the link to the upcoming Blade Servers. Originally by: CCP Wrangler Second, a gentile is a non jewish person
Andrue Amarr	Posted - 2007.09.24 07:06:00 - [4] Unfortunately that kind of virtualisation comes at a price. It allows you to spread a single logical server over multiple physical servers but it does so innefficiently. The only advantage to it (and it is very handy, no denying it) is that it allows you to add more power without the developers having to worry about it. Typically that means you can just add more hardware as and when. Unfortunately you won't get as much out of each piece of new hardware as you would if you customised the software itself to spread across multiple machines. What this means is that if CCP coded Eve to spread systems across multiple nodes they would get better performance than using virtualisation technology to do the same thing. In a large data centre virtualisation is good because the profit margins are higher. You can just charge your customers more and run (I have no real idea of figures) perhpas 20% more machinery than you really need. CCP don't have the luxury of being in that position. They probably have to get more out of each unit than that. -- (Battle hardened industrialist) [Brackley, UK] My budgie can say "ploppy bottom". You have been warned.
Valeo Galaem New Eden Advanced Reconnaissance Unit	Posted - 2007.09.24 08:05:00 - [5] CCP's goal is to upgrade the cluster's network links to InfiniBand within a year. With that they are looking to implement RDMA (remote direct memory access) which will allow one node to share the memory (and therefore processes) of another on the cluster without having to go through the OS first. This way they will be able to get nodes shared over the resources of many computers. When will this happen..? Soon(tm) Thar be Pirates You are not authorised to hack into CONCORD's mainframe Your Wallet has been emptied! CONCORD Encryption Methods
searess	Posted - 2007.09.24 12:45:00 - [6] To the OP , I really dont want to shoot you down in flames here but consider 1 hardware server with say 4 virtual servers on it, Said hardware server we will call master for the object of this excersize. Master server has 8 cpus 32 gig of ram and scsi super fast hard drives. Now master server does work that keeps it at say 70% - 80% utilisation leaving a 20% ish overhead free :) Now bung 4 virtual servers on this master all doing the work of the one master even some more work , seeing as they are virtualised they need to use the master's processing power and memory so your now lookin at pretty much 100% usage. So in conclusion you should not virtualise servers that are working hard already with out suffering degredation in the systems. If you dont beleave me please test this yourself on your virtualised stuff get 3 or 4 hardware servers working at about 80% then virtualise them and dont cheat by virtualising nasty 2 cpu 4 gig things that have been working hard on a monster server that just defeats the object of the excersize. Hijackzorzzzz Sig removed, lacks Eve-related content. If you would like further details please mail [email protected] - Cortes
Firkragg Blue Labs Knights Of the Southerncross	Posted - 2007.09.24 12:59:00 - [7] CCP is moving to a similar technology soon(tm). Atm cpu power is one thing that CCP isnt short of, the cluster is apparently only at about 20% utilisation they just need to redesign the system to be able to make full use of the unused cycles.
Mashie Saldana Minmatar Omerta Syndicate Exuro Mortis	Posted - 2007.09.24 13:15:00 - [8] The problem is latency between the nodes. A virtual system as you suggest will only make it worse if anything. Infiniband and RDMA is one of the few suitable approach for the EVE cluster (and the one planned). Light Assault Launchers & Defender FoF ideas
Ralitge boyter Minmatar	Posted - 2007.09.24 13:54:00 - [9] As stated many times before, yes this works wonders for lets say a web based application. You have lots of requests and thus in most setups lots of threads and thus lots of CPU need but relatively little I/O most web pages do not very hard to share over 4 servers, they all have the same thing to do and do not need to share memory between the boxes. Now look at EVE, you are running one solar system on 4 systems which means all of them need to have local chat in real time, number of members in real time pilots in space their location, activated modules and so on, you are basically sharing a lot of data yet as stated before by a poster EVE in normal operation uses only 20% of it's available CPU cycles. So now you will need a lot more CPU cycles on all 4 boxes to share all the data and you are stressing your memory and available bandwidth between the boxes a lot more just by pushing and shoving all this data around. In the end yes you will win but the investment in power consumption rack space and hardware cost are quite high compared to a situation where CCP codes their server software in such a way that it will natively take care of these things. One could imagine a thread server for local chat the corp. and personal wallet, one for the market and the contracts and one maybe two for the battle space, one for the complexes and so on. This could then be split quite easily to separate servers or CPU's without the need to shove a lot of data about. On top of that if one used infniband instead of GigE you will not tax the CPU cycles to do so and thus will prevent even large chunks of memory moving around from clogging up you CPU with for us gamers useless overhead. As stated before for other types of operations it is great but for such systems as EVE, XEN and VMware are simply not the best posible solution and that is what CCP would like to offer us subscribers to their service. ------------------------------------------- Should you disagree with me, well I guess that is because I disagree with you. If you have a problem with that please feel free not to tell me.
Larshus Magrus Elite Storm Enterprises Storm Armada	Posted - 2007.09.24 14:26:00 - [10] While this is all well and good, and many solutions like this exist in the *nix world, the reality is CCP uses Microsoft server OS packages. The technology for loadbalancing processes over multiple physical machines (OpenMosix, ect al) is far beyond anything that MS has, or will have in the foreseable future. CCP has no intentions to switching to a more mature OS that runs on larger machines, so don't hold your breath for this type of functionality any time soon.
Dr Slaughter Coreli Corporation Exuro Mortis	Posted - 2007.09.24 16:02:00 - [11] Originally by: Larshus Magrus While this is all well and good, and many solutions like this exist in the *nix world, the reality is CCP uses Microsoft server OS packages. The technology for loadbalancing processes over multiple physical machines (OpenMosix, ect al) is far beyond anything that MS has, or will have in the foreseable future. CCP has no intentions to switching to a more mature OS that runs on larger machines, so don't hold your breath for this type of functionality any time soon. Yes, yes, you are 100% correct. It's all Microsofts fault. CCP should use Oracle anyway. I can't understand why they don't. Actually I can't understand why they didn't write the entire application to run on an IBM Z series. OMFG. They're sooooo stupid. Oh, did I already say it's all Microsofts fault? CCP should never have used Microsoft Python. It's so obvious the cluster would have operated better using a mature language like SNOBOL.
Andrue Amarr	Posted - 2007.09.24 17:54:00 - [12] Edited by: Andrue on 24/09/2007 17:54:32 Originally by: Dr Slaughter Originally by: Larshus Magrus While this is all well and good, and many solutions like this exist in the nix world, the reality is CCP uses Microsoft server OS packages. The technology for loadbalancing processes over multiple physical machines (OpenMosix, ect al) is far beyond anything that MS has, or will have in the foreseable future. CCP has no intentions to switching to a more mature OS that runs on larger machines, so don't hold your breath for this type of functionality any time soon. Yes, yes, you are 100% correct. It's all Microsofts fault. CCP should use Oracle anyway. I can't understand why they don't. Actually I can't understand why they didn't write the entire application to run on an IBM Z series. OMFG. They're sooooo stupid. Oh, did I already say it's all Microsofts fault? CCP should never have used Microsoft Python. It's so obvious the cluster would have operated better using a mature language like SNOBOL. Indeed. The poster is obviously determined to get a job in IT by demonstrating his understanding of the situation. All he needs to do now is explain how the world manages to operate so well when over 90% of its computers are running this crappy Microsoft software. Perhaps his explanation will enlighten us as to how come Microsoft is one of the most successful companies on the planet and its founder one of the richest people. I await his explanation with great interest -- (Battle hardened industrialist) [Brackley, UK] My budgie can say "ploppy bottom". You have been warned.*
Sleepkevert Paradox v2.0 Interstellar Alcohol Conglomerate	Posted - 2007.09.24 18:01:00 - [13] Originally by: Popsikle As an example if we start with a test bed of 16 Physical Computers, all interconnected via GigE we can end up with a whole LB'd Web cluster. We can set those 16 servers to act as: 1 Virtual MySQL server (split over 4 physical machines) 4 Virtual Web servers (split over 8 physical machines) 2 Virtual Load balancers (split over 4 physical machines). One word, clusters. Do you really think they have one desktop pc doing all the MSSQL work? They have, 2 or 3 ramsans and the according hardware, EVE solar systems are spread over +25 machines with +50 cpu's, not to mention the two or three redundant proxy's and internet pipes. And i'm guessing that if they can run 30000 simultaneous users in a multi cpu enviroment, they probably have thought of loadbalancing the site a bit.. Please for the love of god, before suggesting this kind of stuff think twice, and read some dev blogs on the subject... Sign my sig
Ryoji Tanakama Caldari Daikoku Fleet Shipyards	Posted - 2007.09.24 18:01:00 - [14] We're currently using Zen to host huge mumbers of linux servers on a small cluster of servers. It was a pain in the ass to get running and we've had no end of silly technical issues resulting in several days of downtime over only a couple of months. I'ts nice on paper but not so great in practice. And what we're doing is about the most simple application of the technology. It is also pretty efficient, though when it does start to grind it doesn't so much creep up on you as it does just stop. ~Ryoji Tanakama Daikoku Fleet Shipyards
JADE DRAG0NESS Dark Scorpions Fate Weavers	Posted - 2007.09.24 18:35:00 - [15] Edited by: JADE DRAG0NESS on 24/09/2007 18:35:01 Originally by: Andrue Edited by: Andrue on 24/09/2007 17:54:32 Originally by: Dr Slaughter Originally by: Larshus Magrus While this is all well and good, and many solutions like this exist in the *nix world, the reality is CCP uses Microsoft server OS packages. The technology for loadbalancing processes over multiple physical machines (OpenMosix, ect al) is far beyond anything that MS has, or will have in the foreseable future. CCP has no intentions to switching to a more mature OS that runs on larger machines, so don't hold your breath for this type of functionality any time soon. Yes, yes, you are 100% correct. It's all Microsofts fault. CCP should use Oracle anyway. I can't understand why they don't. Actually I can't understand why they didn't write the entire application to run on an IBM Z series. OMFG. They're sooooo stupid. Oh, did I already say it's all Microsofts fault? CCP should never have used Microsoft Python. It's so obvious the cluster would have operated better using a mature language like SNOBOL. Indeed. The poster is obviously determined to get a job in IT by demonstrating his understanding of the situation. All he needs to do now is explain how the world manages to operate so well when over 90% of its computers are running this crappy Microsoft software. Perhaps his explanation will enlighten us as to how come Microsoft is one of the most successful companies on the planet and its founder one of the richest people. I await his explanation with great interest Its quite simple really what these companies with big computers do to ensure that Windows works for them is to get out a big pile of cash to be sent to Bill gates and the MS version of the Necromonican and sacrifice a few goats sprinkling the money with the blood before sending it to gates. Seeing that the companies are paying proper homage to the anti-christ [bill gates] he tells the imps that live inside the core of the Windows program that are what really make Windows work to make the big computers work as advertised [sort off]. "Kill one man, and you are a murderer. Kill millions of men, and you are a conqueror. Kill them all, and you are a god." -- Jean Rostand
Alz Shado Ever Flow FREGE Alliance	Posted - 2007.09.24 19:55:00 - [16] Wha? I thought the EVE server was coded in Malbolge.
Dr Slaughter Coreli Corporation Exuro Mortis	Posted - 2007.09.24 20:11:00 - [17] This has been done to death in so many other threads. The stumbling block is that CCP need to improve chunks of their code so it will run across multiple hosts (not just multiple CPUs). It's a stumbling block because it's a huge task. In the mean time they're trying to do everything they can to squeeze more omph out of their existing software architecture. Hardware improvements like infiniband are likely to make minor improvements for us but they will also open up the chance for a major re-write of things like the solar system and grid management code (as someone mentioned features like shared memory make all sorts of things possible). Virtualization is interesting in that you could use features like VmWare's vmotion to move a busy solar system automatically onto bigger hardware (i.e. out of the blade rack and onto the sort of hardware that runs JITA). Would be very cool if they could detect 200 clients setting a destination and pre-load the systems 2 jumps either side on the route onto a JITA type node. Not sure if vmotion would keep up though :) Anyway back to the OP.. sorry I haven't played with XEN beyond what was out there nearly two years ago. My software company develops systems management software for managing VmWare and Microsoft application servers (like SQL, Exchange, etc.) but my background originally was storage managment (sorry if you ever used the first windows release of ARCserve ). Most of the places I see virtualization being used it's for consolidation purposes. Smaller power bills. Less space. More efficient use of tin. Less physical stuff to manage. As someone said above not everything is suitable though.
Andrue Amarr	Posted - 2007.09.24 21:14:00 - [18] Edited by: Andrue on 24/09/2007 21:15:20 Edited by: Andrue on 24/09/2007 21:15:04 Originally by: Dr Slaughter ...my background originally was storage managment (sorry if you ever used the first windows release of ARCserve ). Lol! I used to write data recovery software and although I was on the disk side I have heard several choice comments about ARCserve over the years. Of course they were from people looking at the ugly aftermath of Things(TM) Going Wrong. I had similar comments to make about various file systems 'course none of it compares to the comments I've made over the years about users -- (Battle hardened industrialist) [Brackley, UK] My budgie can say "ploppy bottom". You have been warned.
Popsikle Minmatar Re-Awakened Technologies Inc Electus Matari	Posted - 2007.09.24 23:04:00 - [19] Edited by: Popsikle on 24/09/2007 23:04:53 Originally by: Larshus Magrus While this is all well and good, and many solutions like this exist in the *nix world, the reality is CCP uses Microsoft server OS packages. The technology for loadbalancing processes over multiple physical machines (OpenMosix, ect al) is far beyond anything that MS has, or will have in the foreseable future. CCP has no intentions to switching to a more mature OS that runs on larger machines, so don't hold your breath for this type of functionality any time soon. Windows runs on top of XEN just fine ;) And to the other posters, like I said in my OP I have no real clue as to how they have their setup now other then nodes cant be split among clusters and technology like this was made to be able to handle that. Yes you do lose a portion of the power of each physical component but you gain the ability to scale it without having to modify your code. It would also allow dynamic redistribution of computing power to each cluster as needed versus during downtime which I think would be cool. It definitely is a fun little tech to play around with for web servers and the likes, including game servers like bf/css/ect but no idea the feasibility of it for giant clusters like CCP has setup, nor do I have any in depth exp playing with it yet (as I stated in my OP ;p ) EDIT: Holy run on's batman. ____ <t20> i want to be in a manager potition at Hooters <SaraDawn> Garthagk, do you have it up ? <Garthagk> I can get it up anytime.
Dr Slaughter Coreli Corporation Exuro Mortis	Posted - 2007.09.25 08:37:00 - [20] Originally by: Andrue Edited by: Andrue on 24/09/2007 21:15:20 Edited by: Andrue on 24/09/2007 21:15:04 Originally by: Dr Slaughter ...my background originally was storage managment (sorry if you ever used the first windows release of ARCserve ). Lol! I used to write data recovery software and although I was on the disk side I have heard several choice comments about ARCserve over the years. Of course they were from people looking at the ugly aftermath of Things(TM) Going Wrong. I had similar comments to make about various file systems 'course none of it compares to the comments I've made over the years about users Heh. You should have heard me on the topic of tape drives... We used to send people who hadn't checked to see if their back-ups were write-only to AL downloading, Vogon, or Ontrack.

Pages: [1] :: one page
First page \| Previous page \| Next page \| Last page

COPYRIGHT NOTICE
EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. CCP hf. has granted permission to EVE-Search.com to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Search.com. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this website.