Pages: 1 [2] 3 :: one page |
|
Author |
Thread Statistics | Show CCP posts - 28 post(s) |
Tvashnar Crendraven
The Scope Gallente Federation
4
|
Posted - 2014.08.19 20:57:00 -
[31] - Quote
CCP Goliath wrote:
Bug reports don't get "fixed" per se - they either become or are attached to defects, which are fixed, but there is no way currently to relay the state of the defect to the bug report attached to it. Going back over old bug reports to set them to fixed would be very time consuming.
No doubt. Bugs have costs, and those costs are higher if you ship them to customers.
"'Closed" does not tell me whether I should not be seeing the issue anymore; or whether my understanding is broken and it is not actually a bug or any number of other things which might make me feel that the time I spent to file the bug was not wasted.
Simply closing the issue without comment or a tag like 'fixed', 'not-repro', 'by design' or even 'raised a bug in our real bug tracking system' means that this nice new interface into our bug reports is *useless*. |
Circumstantial Evidence
134
|
Posted - 2014.08.19 21:04:00 -
[32] - Quote
CCP Goliath wrote:Bug reports don't get "fixed" per se - they either become or are attached to defects, which are fixed, but there is no way currently to relay the state of the defect to the bug report attached to it. Going back over old bug reports to set them to fixed would be very time consuming. Please work on automating this :) I have bugs both attached / "I-know-fixed", and attached / "I-don't-know-anything". I would like to see a column with a one word report / state of the internally-written defect our bug is attached to.
|
James Amril-Kesh
4S Corporation Goonswarm Federation
11161
|
Posted - 2014.08.19 21:26:00 -
[33] - Quote
We need to be able to see the resolution, not just "closed". This is an improvement, but not by much. Enjoying the rain today? ;) |
Vincent Athena
V.I.C.E.
2852
|
Posted - 2014.08.19 21:37:00 -
[34] - Quote
I like the idea of being able to vote on legacy bugs. But, if you have no way of setting a bug to at status of "fixed" how will this list be populated? You will not know which bugs to include in the list as not a single one will have any indication that it has been fixed!
I would think that if a bug can be attached to a defect, then when that defect has been fixed, that information could automatically flow back to the bug report.
Also the status "closed" is just not informative enough. If something has "insufficient information", I would like to know, so I can try explaining it again. If it cannot be reproduced, I would like to try again, maybe with some other method for repro that might work for you. If its "working as intended" I would like to know so I can do a feature request in F & I. http://vincentoneve.wordpress.com/ |
|
CCP Habakuk
C C P C C P Alliance
973
|
Posted - 2014.08.19 21:40:00 -
[35] - Quote
Tvashnar Crendraven wrote:CCP Goliath wrote:
Bug reports don't get "fixed" per se - they either become or are attached to defects, which are fixed, but there is no way currently to relay the state of the defect to the bug report attached to it. Going back over old bug reports to set them to fixed would be very time consuming.
No doubt. Bugs have costs, and those costs are higher if you ship them to customers. "'Closed" does not tell me whether I should not be seeing the issue anymore; or whether my understanding is broken and it is not actually a bug or any number of other things which might make me feel that the time I spent to file the bug was not wasted. Simply closing the issue without comment or a tag like 'fixed', 'not-repro', 'by design' or even 'raised a bug in our real bug tracking system' means that this nice new interface into our bug reports is *useless*.
Regarding bugs in "Attached" state: Regardless on what is being shown, you will only know for sure (well in some cases not even then...) that your bug is fixed on TQ, when it will be mentioned in the patch notes (nearly all player-facing bug fixes are mentioned in the patch notes, with the exception of most exploits and some very small issues like spelling fixes). Even if we would show the internal status of the real defect on your bug report it would still cause confusion, as you do not know, when the fix will reach TQ (in some cases this will be the next day, but in some cases it can take a few months (depending in which context it is being fixed) - and no, even the internal information is not as clear.
Regarding bugs in "Closed" state: Old bug reports are unfortunately lacking any further information. I hope that In the future you will often see comments on why a bug report was closed or you will see bug reports with the status "Need more info" with details on what we need. I had a short check at our stats and most bug reports in the "Closed" state were closed because the report was somehow incomplete and the next category are bugs, which were not reproducible. Both cases should improve alot, if we are able to get further information from the reporter. We are unfortunately still not able to send you an email if we need further information, but we are getting closer. We want for sure to give you as much information as easily possible on why your bug report was closed, but we don't know yet on how this will look like exactly when everything is in place.
CCP Habakuk | EVE Quality Assurance | Team Gridlock-á Bug reporting | Mass Testing |
|
|
CCP Habakuk
C C P C C P Alliance
973
|
Posted - 2014.08.19 21:41:00 -
[36] - Quote
Vincent Athena wrote:... Also the status "closed" is just not informative enough. If something has "insufficient information", I would like to know, so I can try explaining it again. If it cannot be reproduced, I would like to try again, maybe with some other method for repro that might work for you. If its "working as intended" I would like to know so I can do a feature request in F & I.
I totally agree and we will be working on improving this. CCP Habakuk | EVE Quality Assurance | Team Gridlock-á Bug reporting | Mass Testing |
|
Logix42
Taxation Damnation
189
|
Posted - 2014.08.19 21:42:00 -
[37] - Quote
Huge thanks to CCP Tier whoever they are! Go beyond the edge of space... Explore |
Hoshi
Brave Newbies Inc. Brave Collective
36
|
Posted - 2014.08.19 22:17:00 -
[38] - Quote
I made a bug report about a month ago about inconsistent use of Agility/Inertia in module descriptions. But the my bug reports page says "You have not submitted any bug reports." (and yes I have checked all account to make sure I didn't submit it on another). Does this mean that there where some error and the bug report never actually reached CCP?. Do I need to resubmit the report?
"Memories are meant to fade. They're designed that way for a reason." |
|
CCP Habakuk
C C P C C P Alliance
978
|
Posted - 2014.08.19 22:23:00 -
[39] - Quote
Hoshi wrote:I made a bug report about a month ago about inconsistent use of Agility/Inertia in module descriptions. But the my bug reports page says "You have not submitted any bug reports." (and yes I have checked all account to make sure I didn't submit it on another). Does this mean that there where some error and the bug report never actually reached CCP?. Do I need to resubmit the report?
I am also not able to find a bug report about this in your given time frame. The issue itself is known and there is an open defect for it, so there is no need to file a new bug report about it. CCP Habakuk | EVE Quality Assurance | Team Gridlock-á Bug reporting | Mass Testing |
|
asteroidjas
Rothschild's Sewage and Septic Sucking Services The Possum Lodge
98
|
Posted - 2014.08.19 22:27:00 -
[40] - Quote
i am currently not able to submit a bug report, as it keeps saying i need to 'fill out all fields' yet gives me no options to select in the 'category' drop-down. |
|
|
CCP Habakuk
C C P C C P Alliance
979
|
Posted - 2014.08.19 23:31:00 -
[41] - Quote
asteroidjas wrote:i am currently not able to submit a bug report, as it keeps saying i need to 'fill out all fields' yet gives me no options to select in the 'category' drop-down.
Hmmm, this is from the client on Tranquility or where do you see this problem? I just tried it and for me it worked fine. It would be helpful if you could get logs from logserver (more info can be found on in the bug reporting page, which is linked in my signature) and create a bug report through the web site. CCP Habakuk | EVE Quality Assurance | Team Gridlock-á Bug reporting | Mass Testing |
|
Noriko Mai
1434
|
Posted - 2014.08.20 00:09:00 -
[42] - Quote
To be honnest, I stopped writing new reports because I thought no one looks into it. After a few months I could not remember what my report was about, just that I've written one. I just checked my bugreports and most of them are attechend to a defect. So someone actually looked at it and something may/will happen. It's is so satisfying. (No joke or sarcasm) *hugs* will write bugreports again. Sorry for not believing |
Kenneth Feld
Habitual Euthanasia Pandemic Legion
106
|
Posted - 2014.08.20 00:16:00 -
[43] - Quote
I submitted a bug in Dec 2013, never heard a thing about it, kinda gave up on bug reports
Looks today and find out nothing has been touched on it. Although i re submitted it on a separate pilot.
23 accounts and 41 pilots, anyway, we could make it so you can see all your bug reports in one area. i found several duplicates today as well, that i didn't even remember writing
Also, added 7 more that i had been putting off writing cause well, not getting feedback kinda leads you to believe no one knows or cares, although seeing one untouched for 9 months kinda does the same thing... |
Salpun
Global Telstar Federation Offices Masters of Flying Objects
767
|
Posted - 2014.08.20 00:51:00 -
[44] - Quote
Currently having to reenter my credentials to access the info pages. And no option to go back once in the page.
If i dont know something about EVE. I check https://wiki.eveonline.com/en/wiki/ISK_The_Guide
See you around the universe. |
|
CCP Habakuk
C C P C C P Alliance
980
|
Posted - 2014.08.20 00:54:00 -
[45] - Quote
Noriko Mai wrote:To be honnest, I stopped writing new reports because I thought no one looks into it. After a few months I could not remember what my report was about, just that I've written one. I just checked my bugreports and most of them are attechend to a defect. So someone actually looked at it and something may/will happen. It's is so satisfying. (No joke or sarcasm) *hugs* will write bugreports again. Sorry for not believing EDIT: I just checked all of them and some are fixed or never made it from SiSi to TQ.
- EBR-15230 is fixed.
- EBR-16144 is fixed.
- EBR-16465 is fixed.
- EBR-19126 is fixed.
So if I'm not mistaken or my reports are only a small part of a bigger defect, all of them can be closed.
I had a short look and it seems like all four of those got fixed because of your bug reports. Thank you!
CCP Habakuk | EVE Quality Assurance | Team Gridlock-á Bug reporting | Mass Testing |
|
|
CCP Habakuk
C C P C C P Alliance
980
|
Posted - 2014.08.20 01:17:00 -
[46] - Quote
Kenneth Feld wrote:I submitted a bug in Dec 2013, never heard a thing about it, kinda gave up on bug reports
Looks today and find out nothing has been touched on it. Although i re submitted it on a separate pilot.
23 accounts and 41 pilots, anyway, we could make it so you can see all your bug reports in one area. i found several duplicates today as well, that i didn't even remember writing
Also, added 7 more that i had been putting off writing cause well, not getting feedback kinda leads you to believe no one knows or cares, although seeing one untouched for 9 months kinda does the same thing...
Yeah, unfortunately we were not successful in being able to process all bug reports within the last year and we did not decide to mass-close the remaining ones (which might have looked better). If you find a very old bug report not closed and you have additional info to it - or you found out that it was not a bug: Please add a comment to it. I'll try to go through bug reports which were updated and see if I can do anything with them.
Btw, some background info: The main part of the bug reports are processed by our volunteer bug hunters and smaller parts are done by us in QA and by our testing partner. Due to several reasons (including the signup being closed for a long time due to technical difficulties) we had less active bug hunters than in previous years - this had quite some impact on our ability to process bug reports. In the last few months the situation improved a lot due to several new and some very active bug hunters (Thanks go especially to BH Nonlocality) - but we could need several additional volunteers. Follow the links in the top post, if you are interested in helping as bug hunter. CCP Habakuk | EVE Quality Assurance | Team Gridlock-á Bug reporting | Mass Testing |
|
Rionan Nafee
Industrie und Handels Konsortium Tribunal Alliance
384
|
Posted - 2014.08.20 07:11:00 -
[47] - Quote
Thank you for your improvements and the infos about them.
One suggestion:
How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?
So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier. |
|
CCP Goliath
C C P C C P Alliance
2364
|
Posted - 2014.08.20 10:12:00 -
[48] - Quote
David Laurentson wrote:CCP Goliath wrote: Thanks very much for telling us how you feel. Sorry that you don't think we take Sisi feedback on board, but I assure you the teams are as diligent as possible when it comes to reading the threads and discussing the ideas in them! That being said, the items in the hypothetical list would already have been acknowledged by us to be issues, so does that go some way to mitigating your concerns?
I think his concern is that those issues were acknowledged when they hit TQ, and closed in error on SiSi. Not ignored in threads, Bug Reports were raised, and closed, despite them actually being bugs. Do you do any investigation of TQ-impacting bugs to see if they were raised on SiSi, and how they were handled? If not, it might be worthwhile: you might have some internal communication work to do, or need to change how you set targets (I can only guess, not working at CCP!).
This might be to do with our version control then. Some bugs will be closed while still being a problem on TQ, because we have verified that the fix works on one of our test environments (Sisi is one, but there are many others)
We do perform RCA (root cause analysis) on certain escaped issues (escaped describes the defect "escaping" from the test environment into the live environment). This is usually facilitated by me, with the involved team(s), and we just go over how the defect managed to get to TQ, strictly looking at the facts and leaving any blame or recrimination out of the picture (one of the worst things you can do, especially in QA, is create an environment where failure has massive negative consequences so that people become afraid of it - that's how problems stay hidden instead of being discussed and solved). Once we know how it happened, we work out what lessons we can take from this so as to not have it happen again, and usually make sure we have a test case or two in our regression suite to cover the defect and areas around it.
Having hard targets with defects is a very double edged sword - some targets are positive, such as "no A or B class bugs on release", others are impossible and dangerous, such as "0 bugs for every feature" or "a good QA should find X bugs per day". CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath |
|
|
CCP Goliath
C C P C C P Alliance
2364
|
Posted - 2014.08.20 10:13:00 -
[49] - Quote
Che Biko wrote:CCP Goliath wrote:Additionally, editing a bug once it has been attached or closed will be futile, as we will no longer be checking that report. Then why not remove the edit option/button for those reports People may not know about the futility, and think "The button/option is still there, so it must serve a purpose."
We likely will in the next iteration, just didn't have time before Tier went on summer vacation CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath |
|
|
CCP Goliath
C C P C C P Alliance
2364
|
Posted - 2014.08.20 10:18:00 -
[50] - Quote
Tvashnar Crendraven wrote:CCP Goliath wrote:
Bug reports don't get "fixed" per se - they either become or are attached to defects, which are fixed, but there is no way currently to relay the state of the defect to the bug report attached to it. Going back over old bug reports to set them to fixed would be very time consuming.
No doubt. Bugs have costs, and those costs are higher if you ship them to customers. "'Closed" does not tell me whether I should not be seeing the issue anymore; or whether my understanding is broken and it is not actually a bug or any number of other things which might make me feel that the time I spent to file the bug was not wasted. Simply closing the issue without comment or a tag like 'fixed', 'not-repro', 'by design' or even 'raised a bug in our real bug tracking system' means that this nice new interface into our bug reports is *useless*.
This seems to be a common sentiment, and we will discuss it internally and see if adding the resolution field (which would show you the Closed resolution, but wouldn't provide any details on fixes, for reasons I've explained in other posts) is something we can fit in to the next iteration. "Attached" means "Raised a bug in our real bug tracking system", relatively speaking. CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath |
|
|
|
CCP Goliath
C C P C C P Alliance
2364
|
Posted - 2014.08.20 10:19:00 -
[51] - Quote
Circumstantial Evidence wrote:CCP Goliath wrote:Bug reports don't get "fixed" per se - they either become or are attached to defects, which are fixed, but there is no way currently to relay the state of the defect to the bug report attached to it. Going back over old bug reports to set them to fixed would be very time consuming. Please work on automating this :) I have bugs both attached / "I-know-fixed", and attached / "I-don't-know-anything". I would like to see a column with a one word report / state of the internally-written defect our bug is attached to.
Impossible without rewriting part of our Jira instance I believe. This isn't a desired solution as it makes plugin maintenance and ongoing support much more complex and risky. Sorry, but at least for now this isn't going to happen. CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath |
|
|
CCP Goliath
C C P C C P Alliance
2364
|
Posted - 2014.08.20 10:24:00 -
[52] - Quote
Vincent Athena wrote:I like the idea of being able to vote on legacy bugs. But, if you have no way of setting a bug to at status of "fixed" how will this list be populated? You will not know which bugs to include in the list as not a single one will have any indication that it has been fixed!
I would think that if a bug can be attached to a defect, then when that defect has been fixed, that information could automatically flow back to the bug report.
Also the status "closed" is just not informative enough. If something has "insufficient information", I would like to know, so I can try explaining it again. If it cannot be reproduced, I would like to try again, maybe with some other method for repro that might work for you. If its "working as intended" I would like to know so I can do a feature request in F & I.
The legacy bugs would be in our database, so would have resolutions, and obviously we can see when defects are fixed - the thing here is that defects are invisible to you, and what you see are called bug reports. The jargon is pretty important when discussing both at the same time to differentiate.
Unfortunately the defect tracking software doesn't work like you describe with the flow of issues, at least at this time, but it does get worked on continually by Atlassian, so who knows what the future may bring!
CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath |
|
|
CCP Goliath
C C P C C P Alliance
2364
|
Posted - 2014.08.20 10:31:00 -
[53] - Quote
Rionan Nafee wrote:Thank you for your improvements and the infos about them.
One suggestion:
How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?
So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier.
This is something that's come up in our meetings before. On the one hand, as you say it's great to give people positive feedback when they submit a report that leads to us fixing or discovering an issue. On the other hand, the maintenance around such a system (you'd need rules, and checks, and consistency, etc) as well as the work that would have to go into creating it, might be a little much, or the wrong effort in the wrong place... It's definitely not entirely ruled out, but right now I don't have a good idea of how it would work well either. CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath |
|
Momiji Sakora
Omni Galactic
23
|
Posted - 2014.08.20 10:44:00 -
[54] - Quote
This is great progress, been asking and wanting something like this for a long while. So glad to see it actually implemented!
I would like a bit of feedback in the general "This was closed because XYZ" or "Resolved in Hyperion build #####" etc. I don't think we need personalised responses of course, that'd be a bit too much.
EDIT: I can see why this might not be implemented anytime soon, but it would be a welcome addition. Generally when working with Jira we've merged duplicates and then used the link option to link it to duplicates. It'd just require a copy paste fix, obviously it's working out if the time would be better spent elsewhere. |
Rionan Nafee
Industrie und Handels Konsortium Tribunal Alliance
384
|
Posted - 2014.08.20 16:11:00 -
[55] - Quote
CCP Goliath wrote:Rionan Nafee wrote:Thank you for your improvements and the infos about them.
One suggestion:
How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?
So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier. This is something that's come up in our meetings before. On the one hand, as you say it's great to give people positive feedback when they submit a report that leads to us fixing or discovering an issue. On the other hand, the maintenance around such a system (you'd need rules, and checks, and consistency, etc) as well as the work that would have to go into creating it, might be a little much, or the wrong effort in the wrong place... It's definitely not entirely ruled out, but right now I don't have a good idea of how it would work well either. Thank you for this information.
Is there any other way possible to know if the submitted bug reports are usefull or only a pain in the ass for the bug hunters? |
|
CCP Goliath
C C P C C P Alliance
2365
|
Posted - 2014.08.20 16:44:00 -
[56] - Quote
Rionan Nafee wrote:CCP Goliath wrote:Rionan Nafee wrote:Thank you for your improvements and the infos about them.
One suggestion:
How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?
So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier. This is something that's come up in our meetings before. On the one hand, as you say it's great to give people positive feedback when they submit a report that leads to us fixing or discovering an issue. On the other hand, the maintenance around such a system (you'd need rules, and checks, and consistency, etc) as well as the work that would have to go into creating it, might be a little much, or the wrong effort in the wrong place... It's definitely not entirely ruled out, but right now I don't have a good idea of how it would work well either. Thank you for this information. Is there any other way possible to know if the submitted bug reports are usefull or only a pain in the ass for the bug hunters?
At this time there definitely isn't an easy way to relay that info back to you, barring commenting "great report" or similar on the report, which will absolutely have to be left up to the individuals processing the reports (but will be encouraged) CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath |
|
Salpun
Global Telstar Federation Offices Masters of Flying Objects
767
|
Posted - 2014.08.20 16:47:00 -
[57] - Quote
CCP Goliath wrote:Rionan Nafee wrote:CCP Goliath wrote:Rionan Nafee wrote:Thank you for your improvements and the infos about them.
One suggestion:
How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?
So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier. This is something that's come up in our meetings before. On the one hand, as you say it's great to give people positive feedback when they submit a report that leads to us fixing or discovering an issue. On the other hand, the maintenance around such a system (you'd need rules, and checks, and consistency, etc) as well as the work that would have to go into creating it, might be a little much, or the wrong effort in the wrong place... It's definitely not entirely ruled out, but right now I don't have a good idea of how it would work well either. Thank you for this information. Is there any other way possible to know if the submitted bug reports are usefull or only a pain in the ass for the bug hunters? At this time there definitely isn't an easy way to relay that info back to you, barring commenting "great report" or similar on the report, which will absolutely have to be left up to the individuals processing the reports (but will be encouraged)
Even showing how many comments have been attached to the reports would be enough to show that the items is getting worked on. Having that number in the main bugreport list would save the bandwitdh required for people to check their bug reports for comments.
If i dont know something about EVE. I check https://wiki.eveonline.com/en/wiki/ISK_The_Guide
See you around the universe. |
Soldarius
Deadman W0nderland Test Alliance Please Ignore
763
|
Posted - 2014.08.20 18:55:00 -
[58] - Quote
http://content.eveonline.com/www/newssystem/media/66330/1/bugreportpic1.png
Confirming that TEST has won Eve. Either that, or TEST is a bug and should be... corrected. You decide.
Also, cool stuff. Knowing where you're bur report is in the system is a good thing. People like feedback and knowing that they have been heard. GÇ£I personally refuse to help AAA take space from itself so it can become an even shittier version of itselfGÇ¥ -Grath Telkin, 2014.
Free PASTA! |
Kenneth Feld
Habitual Euthanasia Pandemic Legion
107
|
Posted - 2014.08.22 02:31:00 -
[59] - Quote
well, I got my bug report "Attached" but attached to what? EBR-22458 What is going to get done?
The bug report is kinda one of those it used to be this way, now it is that way...
Attached doesn't tell me if the old way is correct or the new way is correct.....
How the heck am i supposed to know whether to shut up and wait 6 years for it to revert to the old way or 4 years for it to be fixed the new way?
At this rate, going back to old style forum whining is still the best answer...... |
|
CCP Goliath
C C P C C P Alliance
2367
|
Posted - 2014.08.22 09:50:00 -
[60] - Quote
Kenneth Feld wrote:well, I got my bug report "Attached" but attached to what? EBR-22458 What is going to get done?
The bug report is kinda one of those it used to be this way, now it is that way...
Attached doesn't tell me if the old way is correct or the new way is correct.....
How the heck am i supposed to know whether to shut up and wait 6 years for it to revert to the old way or 4 years for it to be fixed the new way?
At this rate, going back to old style forum whining is still the best answer......
Attached means that it has either had a defect created from it, or attached to an existing defect we already had. This is basically the best outcome of a bug report. As I've explained further up the thread, there is no way for us to currently relay the status of defects that bug reports are attached to, to you. In the case of this issue, you filed it on the 19th August, we processed it on the 21st and CCP Claymore made a defect from it. That defect is now assigned to a developer and waiting to be fixed (note, I do not know the nature of the intended fix). If the report was invalid, it would show as Closed rather than Attached. CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath |
|
|
|
|
|
Pages: 1 [2] 3 :: one page |
First page | Previous page | Next page | Last page |