Solving Partial Differential Equations (PDEs) is fundamental to understanding various physical phenomena across various disciplines. Traditional solvers in industry often face challenges such as computational complexity, mesh generation, and convergence issues. In recent years, neural networks have emerged as a promising alternative for tackling PDEs due to their ability to approximate complex functions and learn underlying patterns from data. This thesis presents novel advancements in solving PDEs using neural networks, coupled with innovative approaches in network initialization and domain decomposition techniques. Firstly, a neural network-based solver is introduced to solve PDEs efficiently over square domains. Employing a neural network architecture with a single hidden layer, emphasis is placed on training only the final layer to directly solve PDEs without the need for explicit data. This methodology handles non-time-dependent PDEs with boundary conditions, including linear and non-linear equations. Secondly, the thesis explores domain decomposition strategies combined with neural networks for tackling complex PDE problems. By partitioning the domain into subdomains, each has its neural network approximator. The coupling of neural networks at shared boundary points ensures solution continuity over the entire domain and accuracy in the overall solution. Furthermore, we dive into the "Sampling Where It Matters" (SWIM) concept, aimed at enhancing the initialization of neural networks for improved performance in various tasks, including PDE solving. SWIM leverages the gradient of the truth function along with input points to intelligently initialize network parameters, leading to more efficient convergence and enhanced solution accuracy. Through several experiments, the efficacy and versatility of the proposed methodologies are demonstrated, including a comparative study against existing methods such as Physics-Informed Neural Network (PINN) and Finite Element Method (FEM). This comparative analysis showcases where the presented method stands against existing methods, providing valuable insights into their performance and applicability.