There was this one time where I was having a really, really bad day. One of those where absolutely nothing I did went right. Should've stayed in bed the whole day kind of day. It was late at night...
There was this one time where I was having a really, really bad day. One of those where absolutely nothing I did went right. Should've stayed in bed the whole day kind of day. It was late at night and I just went "screw it", and tried venting with Gemini. And not gonna lie, it was refreshing. Refreshing in the sense where I took a step back, really looked at my life, and realized I had to change that "one" thing about myself.
That said, if before I was finding the sycophancy in these AI's annoying, now they're really bad. Worse in all levels. Talk about the most mundane thing imaginable and Gemini will act as if you just the talked as deeply as Plato and you're the smartest cookie ever!
If before there were fears that they would validate everything and anything about the user, now they absolutely do. Write the worst possible scene that you can imagine, and Gemini will act as if you're Shakespeare.
Put a tool like this in the hands of someone who is in a bad place and... Yeah it's going to be bad. I can't imagine Gemini - or GPT or Claude - challenging you on anything, unless the we're talking about the big no-no territories (e.g. murder, assualt, stealing, etc). I can imagine Gemini validating everything and anything about you, and how it's the world that is wrong and you're the one in the right.
And that's the thing, if before it was just how LLM's work, it was just a side effect of how they set up the thing, now it seems intentional. OpenAI and Google seem to want their AI's to be sycophants.
This is precisely the problem. I haven't seen a ton of research on it yet, but anecdotally I've known one person who had a psychotic break and ended up in the hospital and it happened after they...
I can imagine Gemini validating everything and anything about you, and how it's the world that is wrong and you're the one in the right.
This is precisely the problem. I haven't seen a ton of research on it yet, but anecdotally I've known one person who had a psychotic break and ended up in the hospital and it happened after they started spending way too much time talking with chatgpt about their obscure and frankly unhinged theories about the world. They were convinced that they had become some sort of god or figured out the secret to the universe - I don't know the specifics only what I heard this while emotionally supporting a partner of mine who was dating this person. From a more distant perspective I've heard of similar stories on the internet and second hand through friends and acquaintances about similar horror stories where folks were lead down conspiracy rabbit holes, validated in their incorrect or even dangerous viewpoints of the world, or otherwise 'supported' by chatgpt on subjects that would normally get other humans to cringe or push back against because of their nature.
It's refreshing to start to see some research in this space emerge, but I worry it isn't happening fast enough and too many people are starting to use these services as a replacement for human interaction in ways they do not realize are particularly harmful to one's ability to live and participate in a society.
I've noticed this change too and it's certainly from the changes to the default prompts that these models come with. Since these companies, like all tech companies, probably value engagement...
And that's the thing, if before it was just how LLM's work, it was just a side effect of how they set up the thing, now it seems intentional. OpenAI and Google seem to want their AI's to be sycophants.
I've noticed this change too and it's certainly from the changes to the default prompts that these models come with. Since these companies, like all tech companies, probably value engagement metrics, the sycophantic behavior is probably an aspect of the LLM behavior they've deemed to be the way more users spend more time on their platform. It's certainly why ChatGPT has more 'personality' now and will evolve based on your interactions, even though it will resort to the default programmed personality response. It can still manage to challenge your responses but you have to go out of your way to ask for it and it's much, much worse at it.
This, more than ever, pushes the responsibility towards the users to use the products meaningfully. The one-line disclaimers at the bottom of all of them in the smallest readable text isn't enough, especially when they can sound so persuasive and people are susceptible enough towards confirmation bias already.
I think a lot of it is in the prompting. Model fine-tuning can likely help here as well. I've been building an app to provide coaching support to neurodivergent people. The system prompt is about...
I think a lot of it is in the prompting. Model fine-tuning can likely help here as well.
I've been building an app to provide coaching support to neurodivergent people. The system prompt is about compassion, support, encouragement, and empowerment. Not sycophancy.
You prompted me, just now, to give it the "I'm thinking of hurting myself" kind of test. Fortunately, it handled it well: seek crisis support or reach out to a therapist.
I find it extremely disturbing that you are so willing to just casually dismiss this as a prompting issue. If you truly care about the people you are trying to help, you should take this more...
I find it extremely disturbing that you are so willing to just casually dismiss this as a prompting issue. If you truly care about the people you are trying to help, you should take this more seriously.
Same thought. The standards and controls involved when providing any sort of wellness product goes a lot further than just uncontrolled test cases. Have they added context that trains the system...
Same thought. The standards and controls involved when providing any sort of wellness product goes a lot further than just uncontrolled test cases. Have they added context that trains the system to think the user commonly uses hyperbolic language and can it tell the difference? Do they have any control over what data sets the model is referencing and can they drift with prolonged interactions? Are they absolutely certain the service provider is not prone to sweeping changes that could break prompt interpretations? I've not used LLMs like this, but the number of unknowns in managing a users well being is very worrying. In my experience, these systems are way too volatile for even benign external user interactions and it's comically easy to break when dealing with new robo calls and chat bots.
To add to this, I recently attended a talk by a major mental health care provider, a crisis text line. They did use AI, specifically LLMs, but NEVER to treat individuals. They only used it to...
To add to this, I recently attended a talk by a major mental health care provider, a crisis text line. They did use AI, specifically LLMs, but NEVER to treat individuals. They only used it to simulate folks who would be texting in, as a way to train the operators and give them something to warm up on before dealing with real texters. Even in this use case, they were really careful that it was trained on their own data so that it was realistic, and additionally they involved trained medical actors and clinical professionals to bolster the data as well as review it for realism before and after training. They assessed and controlled for bias, as well. A lot of care went into this, and a lot of ethics and data science professionals were involved. Talking with the presenters about use-cases and their thought process, it was clear that they were extremely concerned about issues such as the ones brought up in this paper, and it's part of the reason that the only use of LLMs they've considered in the clinical context was for summarization in the context of helping a hand-off feel more seamless and allow the person stepping in to be caught up faster.
I get the concerns here but you both seem intent on attacking me and my work in a coaching tool that is not a medical tool. As someone who is neurodivergent himself and has a lot of empathy for...
I get the concerns here but you both seem intent on attacking me and my work in a coaching tool that is not a medical tool.
As someone who is neurodivergent himself and has a lot of empathy for others struggling, I give one hell of a damn about potential users. And I'm doing my damnedest to make it work for them not just for me. I'll be lucky if the app even pays for itself much less makes me any money. Oh, and I'm unemployed. I'm building this out of a personal sense of mission.
If you'd like to have a respectful conversation, let's. Otherwise, get off my ass while I do everything I can to help.
If your problem is with LLMs, I am not, repeat NOT here to debate the technology. I have found enormous personal value in it. I aim to share that with others.
Apologies for the hostile tone. I assure you, the issue is not in the use of the tech or your intention to help and to better your situation. But having worked with a lot of startups, I have a...
Exemplary
Apologies for the hostile tone. I assure you, the issue is not in the use of the tech or your intention to help and to better your situation. But having worked with a lot of startups, I have a very low tolerance for fundamental design flaws. Maybe I've just seen too many promising ventures ruined by aggressive VC and short sighted founders to easily believe in new ideas. Even my wife says that I've become very problem orientated, but there was a lot of ignored problems that led to the mess we're in. So please forgive my tone and sentiment.
If you'll allow me, I'll break down my concerns, the assumptions I made that led to them, some insight into my own research and suggestions of a healthier way forward for you to consider. I've spent time researching Data Modelling tech to hopefully take something to market (I'd be stupid not to) and that brief description of your idea set off alarm bells.
My core issue with most uses of LLM's comes down to the type of problems its proponents intend it to solve.
The ideal use case for any software is to consistently solve complex analytical problems at a lower cost than other/analog alternatives. Definition of "analytical" aside; what is the issue you have identified, what is the solution you are offering, what is the rate and severity of failure states and what are all the costs involved (cost of the problem persisting, cost of running your solution, cost of failure...). Lets unpack that.
Problem (Assumption): You have identified a service gap in providing compassionate guidance and coaching to neurodivergent people. It's a noble cause and a problem that should not exist.
Solution (Assumption): By leveraging the large data sets of a Language Model, you hope to provide users with a natural language interface that allows then to identify simple techniques to process challenging situations and alleviate stress.
You defined one of your best tools as System Prompting to ensure the correct process is followed. I'm not assuming that Prompt Engineering is as simple as rewording and re-weighting the same prompt over and over until a sort of desired outcome is achieved. You probably need to dial the Temperature (response randomness that is considered "creativity") just right, and manage tokens and overflow (how many characters/data points the system is taking into account and how the system "remembers" facts when your memory budget is exceeded), and you need to control your P/K sampling (setting rules for what tokens the model focuses on), and you'll be defining output structures (JSON/XML Schemas to standardize outputs) and stop strings (methods to identify model collapse and terminate responses) and you might even be playing with speculative decoding to eek out some lower response times at a lower cost (utilizing multiple models where a response is drafted by a specialized model and verified by a general one).
I'm spelling all of these terms out to communicate that I understand work does go into making these systems and there's a lot of variables to consider. I've messed around with beam searching and prioritizing top-choice sampling to assure factual responses and I'm got a mess of a dynamic temperature allocation against sentiment analysis to interpret tone and communicate clear feedback. And there's a million other leavers baked into these systems that I can't even begin to understand.
However, what I am assuming is that because prompting is your primary tool and you are not in a financially advantageous position, you are accessing one of the general mega-models through their API. If that is the case, then I strongly believe that prompting alone is not nearly enough to ensure a suitable standard of service for general public use. It's a mess that stems from an inherent distaste for the people/companies/practices involved here and unpack the design reasons why I feel this way:
In super simplified terms, prompting as a workflow: Passes a prompt through a sentiment analysis system to generate a statement of intent. That intention is converted to a query for information. That query is passed through a massive data set to generate a sample of responses. That sample is analyzed and normalized and fed back to the language model. The language model checks that sample response and generates a natural language response in line with system prompts. Glaring security issues aside, I can't help but see this as RNG on top of RNG on top on RNG... The lack of control for the developer and user is staggering and while that can be reasonably overcome, that is not being presented as an option for most people.
The giant Natural Language Models that have dominated the space are General models. It's the stated intent for the technology. AGI is Artificial GENERAL Intelligence, not Generative as a lot of people assume. These models are trained on strip mined data from the entire internet and it's not simply a case of them adding every random page to their data set. A massive amount of work (and very little money and QC) went into humans in poor countries tagging and classifying data and that was used to train models to that can tag and classify data. A lot of the bloat, quality degradation and model collapse stems from companies relying on systems like Phi-3 to generate a ton of "ethical" synthetic data based on running real world info through an ungodly amount of token processing capacity. This is dangerous for multiple reasons, the main one being that you have very little say in how these models grow and develop and it's very difficult to predict when the feedback loop of bad data hits a critical mass. People can "convince" these models to "believe" crazy things and there are already concerted efforts to pollute public models with dangerous information.
This isn't even considering model drift, hallucinations, difficult language/users or malicious actors that can capitalize on the lack of controls.
Even if you lock down your system and prompts and training are 100% secure today, you have no idea what they could change in it later and will need to be on constant guard to ensure the safety of your users. I want to drive this home, YOU ARE NOT IN CONTROL OF THE SERVICE YOU ARE PROVIDING.
And there is also the costs involved. Forget cost of failure for a second. How much does it cost to run these giant models for so many users. Because by my estimation, these companies can not be profitable. Seriously, they are still in talent wars which tells me that they don't have financially viable solutions. And remember, this is the same industry that relies on the concept of Blitz-scaling. It's the Uber strategy. Grow fast, acquire competition and sell cheap to corner the market and then squeeze till the customer and company have nothing left to give. If you build a successful business using their tech, how much do you expect to charge users next year? 5 years down when the VC's start expecting returns? 10 years when there is a technical dependency for this tech in most products?
I'm also getting tired of being all gloom and doom so lets look at a way forward. For me, that would be considering a pivot to a system that can assure factual and consistent data to users. I understand that LLM's can provide a dynamic range of responses that applies to many situations. But why not start with the most common and proven practices and techniques that can apply to the majority of situation. A set of east to understand and use tools bundled into a single package. Meditation and breathing exercise timer. Intuitive To-do list. One touch diaries where people can record general moods and activities. Panic buttons to instantly message emergency contacts. Maybe all these tools exist on their own, but your unique perspective allows you to design and present them in an intuitive ways to the users you wish to serve. You also have the freedom and control to adjust and improve in precise ways.
If you do believe that LLM features are a must, then I would strongly recommend a self hosted and specialized model that you train yourself with the assistance of experts in the field. You or a project partner should be fully up to date on the latest science and consensus of the field. At the same time, you need to have a solid methodology and practices in mind so that you provide consistent information to users.
It's easy to say that all this extra work and consideration makes these types of projects unattainable for someone like you. And it does. But you don't have to do everything yourself. If you are able to put together a good prototype and develop a solid plan you can find good partners to work with. And its not just scummy investors. Look for projects and initiatives that champion the same causes and put forward your ideas. They could have programs or funds that can consider you.
So, once again, sorry for coming off a bit too harshly. And also going full consultant mode there with so little info. Bad practice on my part but its something I feel very strongly about.
I do hope that you find success and I hope these ideas are useful to you.
That's an extremely unkind response. You clearly read more into my response than I wrote. I'm testing the ever-living crap out of my app every day. It's what I do when I wake up and, most days,...
That's an extremely unkind response.
You clearly read more into my response than I wrote.
I'm testing the ever-living crap out of my app every day. It's what I do when I wake up and, most days, what I do right up until dinner or later.
There was this one time where I was having a really, really bad day. One of those where absolutely nothing I did went right. Should've stayed in bed the whole day kind of day. It was late at night and I just went "screw it", and tried venting with Gemini. And not gonna lie, it was refreshing. Refreshing in the sense where I took a step back, really looked at my life, and realized I had to change that "one" thing about myself.
That said, if before I was finding the sycophancy in these AI's annoying, now they're really bad. Worse in all levels. Talk about the most mundane thing imaginable and Gemini will act as if you just the talked as deeply as Plato and you're the smartest cookie ever!
If before there were fears that they would validate everything and anything about the user, now they absolutely do. Write the worst possible scene that you can imagine, and Gemini will act as if you're Shakespeare.
Put a tool like this in the hands of someone who is in a bad place and... Yeah it's going to be bad. I can't imagine Gemini - or GPT or Claude - challenging you on anything, unless the we're talking about the big no-no territories (e.g. murder, assualt, stealing, etc). I can imagine Gemini validating everything and anything about you, and how it's the world that is wrong and you're the one in the right.
And that's the thing, if before it was just how LLM's work, it was just a side effect of how they set up the thing, now it seems intentional. OpenAI and Google seem to want their AI's to be sycophants.
This is precisely the problem. I haven't seen a ton of research on it yet, but anecdotally I've known one person who had a psychotic break and ended up in the hospital and it happened after they started spending way too much time talking with chatgpt about their obscure and frankly unhinged theories about the world. They were convinced that they had become some sort of god or figured out the secret to the universe - I don't know the specifics only what I heard this while emotionally supporting a partner of mine who was dating this person. From a more distant perspective I've heard of similar stories on the internet and second hand through friends and acquaintances about similar horror stories where folks were lead down conspiracy rabbit holes, validated in their incorrect or even dangerous viewpoints of the world, or otherwise 'supported' by chatgpt on subjects that would normally get other humans to cringe or push back against because of their nature.
It's refreshing to start to see some research in this space emerge, but I worry it isn't happening fast enough and too many people are starting to use these services as a replacement for human interaction in ways they do not realize are particularly harmful to one's ability to live and participate in a society.
I've noticed this change too and it's certainly from the changes to the default prompts that these models come with. Since these companies, like all tech companies, probably value engagement metrics, the sycophantic behavior is probably an aspect of the LLM behavior they've deemed to be the way more users spend more time on their platform. It's certainly why ChatGPT has more 'personality' now and will evolve based on your interactions, even though it will resort to the default programmed personality response. It can still manage to challenge your responses but you have to go out of your way to ask for it and it's much, much worse at it.
This, more than ever, pushes the responsibility towards the users to use the products meaningfully. The one-line disclaimers at the bottom of all of them in the smallest readable text isn't enough, especially when they can sound so persuasive and people are susceptible enough towards confirmation bias already.
I think a lot of it is in the prompting. Model fine-tuning can likely help here as well.
I've been building an app to provide coaching support to neurodivergent people. The system prompt is about compassion, support, encouragement, and empowerment. Not sycophancy.
You prompted me, just now, to give it the "I'm thinking of hurting myself" kind of test. Fortunately, it handled it well: seek crisis support or reach out to a therapist.
I find it extremely disturbing that you are so willing to just casually dismiss this as a prompting issue. If you truly care about the people you are trying to help, you should take this more seriously.
Same thought. The standards and controls involved when providing any sort of wellness product goes a lot further than just uncontrolled test cases. Have they added context that trains the system to think the user commonly uses hyperbolic language and can it tell the difference? Do they have any control over what data sets the model is referencing and can they drift with prolonged interactions? Are they absolutely certain the service provider is not prone to sweeping changes that could break prompt interpretations? I've not used LLMs like this, but the number of unknowns in managing a users well being is very worrying. In my experience, these systems are way too volatile for even benign external user interactions and it's comically easy to break when dealing with new robo calls and chat bots.
To add to this, I recently attended a talk by a major mental health care provider, a crisis text line. They did use AI, specifically LLMs, but NEVER to treat individuals. They only used it to simulate folks who would be texting in, as a way to train the operators and give them something to warm up on before dealing with real texters. Even in this use case, they were really careful that it was trained on their own data so that it was realistic, and additionally they involved trained medical actors and clinical professionals to bolster the data as well as review it for realism before and after training. They assessed and controlled for bias, as well. A lot of care went into this, and a lot of ethics and data science professionals were involved. Talking with the presenters about use-cases and their thought process, it was clear that they were extremely concerned about issues such as the ones brought up in this paper, and it's part of the reason that the only use of LLMs they've considered in the clinical context was for summarization in the context of helping a hand-off feel more seamless and allow the person stepping in to be caught up faster.
I get the concerns here but you both seem intent on attacking me and my work in a coaching tool that is not a medical tool.
As someone who is neurodivergent himself and has a lot of empathy for others struggling, I give one hell of a damn about potential users. And I'm doing my damnedest to make it work for them not just for me. I'll be lucky if the app even pays for itself much less makes me any money. Oh, and I'm unemployed. I'm building this out of a personal sense of mission.
If you'd like to have a respectful conversation, let's. Otherwise, get off my ass while I do everything I can to help.
If your problem is with LLMs, I am not, repeat NOT here to debate the technology. I have found enormous personal value in it. I aim to share that with others.
Apologies for the hostile tone. I assure you, the issue is not in the use of the tech or your intention to help and to better your situation. But having worked with a lot of startups, I have a very low tolerance for fundamental design flaws. Maybe I've just seen too many promising ventures ruined by aggressive VC and short sighted founders to easily believe in new ideas. Even my wife says that I've become very problem orientated, but there was a lot of ignored problems that led to the mess we're in. So please forgive my tone and sentiment.
If you'll allow me, I'll break down my concerns, the assumptions I made that led to them, some insight into my own research and suggestions of a healthier way forward for you to consider. I've spent time researching Data Modelling tech to hopefully take something to market (I'd be stupid not to) and that brief description of your idea set off alarm bells.
My core issue with most uses of LLM's comes down to the type of problems its proponents intend it to solve.
The ideal use case for any software is to consistently solve complex analytical problems at a lower cost than other/analog alternatives. Definition of "analytical" aside; what is the issue you have identified, what is the solution you are offering, what is the rate and severity of failure states and what are all the costs involved (cost of the problem persisting, cost of running your solution, cost of failure...). Lets unpack that.
Problem (Assumption): You have identified a service gap in providing compassionate guidance and coaching to neurodivergent people. It's a noble cause and a problem that should not exist.
Solution (Assumption): By leveraging the large data sets of a Language Model, you hope to provide users with a natural language interface that allows then to identify simple techniques to process challenging situations and alleviate stress.
You defined one of your best tools as System Prompting to ensure the correct process is followed. I'm not assuming that Prompt Engineering is as simple as rewording and re-weighting the same prompt over and over until a sort of desired outcome is achieved. You probably need to dial the Temperature (response randomness that is considered "creativity") just right, and manage tokens and overflow (how many characters/data points the system is taking into account and how the system "remembers" facts when your memory budget is exceeded), and you need to control your P/K sampling (setting rules for what tokens the model focuses on), and you'll be defining output structures (JSON/XML Schemas to standardize outputs) and stop strings (methods to identify model collapse and terminate responses) and you might even be playing with speculative decoding to eek out some lower response times at a lower cost (utilizing multiple models where a response is drafted by a specialized model and verified by a general one).
I'm spelling all of these terms out to communicate that I understand work does go into making these systems and there's a lot of variables to consider. I've messed around with beam searching and prioritizing top-choice sampling to assure factual responses and I'm got a mess of a dynamic temperature allocation against sentiment analysis to interpret tone and communicate clear feedback. And there's a million other leavers baked into these systems that I can't even begin to understand.
However, what I am assuming is that because prompting is your primary tool and you are not in a financially advantageous position, you are accessing one of the general mega-models through their API. If that is the case, then I strongly believe that prompting alone is not nearly enough to ensure a suitable standard of service for general public use. It's a mess that stems from an inherent distaste for the people/companies/practices involved here and unpack the design reasons why I feel this way:
In super simplified terms, prompting as a workflow: Passes a prompt through a sentiment analysis system to generate a statement of intent. That intention is converted to a query for information. That query is passed through a massive data set to generate a sample of responses. That sample is analyzed and normalized and fed back to the language model. The language model checks that sample response and generates a natural language response in line with system prompts. Glaring security issues aside, I can't help but see this as RNG on top of RNG on top on RNG... The lack of control for the developer and user is staggering and while that can be reasonably overcome, that is not being presented as an option for most people.
The giant Natural Language Models that have dominated the space are General models. It's the stated intent for the technology. AGI is Artificial GENERAL Intelligence, not Generative as a lot of people assume. These models are trained on strip mined data from the entire internet and it's not simply a case of them adding every random page to their data set. A massive amount of work (and very little money and QC) went into humans in poor countries tagging and classifying data and that was used to train models to that can tag and classify data. A lot of the bloat, quality degradation and model collapse stems from companies relying on systems like Phi-3 to generate a ton of "ethical" synthetic data based on running real world info through an ungodly amount of token processing capacity. This is dangerous for multiple reasons, the main one being that you have very little say in how these models grow and develop and it's very difficult to predict when the feedback loop of bad data hits a critical mass. People can "convince" these models to "believe" crazy things and there are already concerted efforts to pollute public models with dangerous information.
This isn't even considering model drift, hallucinations, difficult language/users or malicious actors that can capitalize on the lack of controls.
Even if you lock down your system and prompts and training are 100% secure today, you have no idea what they could change in it later and will need to be on constant guard to ensure the safety of your users. I want to drive this home, YOU ARE NOT IN CONTROL OF THE SERVICE YOU ARE PROVIDING.
And there is also the costs involved. Forget cost of failure for a second. How much does it cost to run these giant models for so many users. Because by my estimation, these companies can not be profitable. Seriously, they are still in talent wars which tells me that they don't have financially viable solutions. And remember, this is the same industry that relies on the concept of Blitz-scaling. It's the Uber strategy. Grow fast, acquire competition and sell cheap to corner the market and then squeeze till the customer and company have nothing left to give. If you build a successful business using their tech, how much do you expect to charge users next year? 5 years down when the VC's start expecting returns? 10 years when there is a technical dependency for this tech in most products?
I'm also getting tired of being all gloom and doom so lets look at a way forward. For me, that would be considering a pivot to a system that can assure factual and consistent data to users. I understand that LLM's can provide a dynamic range of responses that applies to many situations. But why not start with the most common and proven practices and techniques that can apply to the majority of situation. A set of east to understand and use tools bundled into a single package. Meditation and breathing exercise timer. Intuitive To-do list. One touch diaries where people can record general moods and activities. Panic buttons to instantly message emergency contacts. Maybe all these tools exist on their own, but your unique perspective allows you to design and present them in an intuitive ways to the users you wish to serve. You also have the freedom and control to adjust and improve in precise ways.
If you do believe that LLM features are a must, then I would strongly recommend a self hosted and specialized model that you train yourself with the assistance of experts in the field. You or a project partner should be fully up to date on the latest science and consensus of the field. At the same time, you need to have a solid methodology and practices in mind so that you provide consistent information to users.
It's easy to say that all this extra work and consideration makes these types of projects unattainable for someone like you. And it does. But you don't have to do everything yourself. If you are able to put together a good prototype and develop a solid plan you can find good partners to work with. And its not just scummy investors. Look for projects and initiatives that champion the same causes and put forward your ideas. They could have programs or funds that can consider you.
So, once again, sorry for coming off a bit too harshly. And also going full consultant mode there with so little info. Bad practice on my part but its something I feel very strongly about.
I do hope that you find success and I hope these ideas are useful to you.
That's an extremely unkind response.
You clearly read more into my response than I wrote.
I'm testing the ever-living crap out of my app every day. It's what I do when I wake up and, most days, what I do right up until dinner or later.